Tag Archives: php

php tail

I have a php script that frequently needs to email me the last few lines of a log file. I can’t afford to exec() a binary tail process, so the solution has to be in pure php.

Originally, the files in question never exceeded more than a few thousand lines. Unfortunately, I am encountering cases now where the files are now occasionally 50,000 lines or longer. This causes PHP’s memory consumption to explode.

Note: Code snippets provided here are not fully functional standalone shell scripts. The scripts I ran to benchmark the algorithms contain some rudimentary setup logic that is not important here, so has not been included.

My original method:

This is easy to understand and is pretty fast, all things considered. Unfortunately, the memory footprint for loading a file into an array is obscene. Loading a 4400 line log file with this method could consume more than 17mb of ram. 50,000 line files easily stressed the 256mb limit I am able to provide the process.

So, the obvious solution to the memory consumption is to avoid loading the entire file at once. What if we kept a rotating list of lines in the file?

This method works by keeping the $lines-many most recent lines of the file in an array. Memory consumption remains sane, but the performance hit for performing so many array pushes and shifts is bad. Really bad. With small files, I can’t notice any difference between this method and the file() method… but with longer files, it adds up quickly.

Given a 51 line, 4kb file, an average execution ($lines = 20) might look like this:
[code]
ammon@zapp:~$ time ./tail-file.php a.log >/dev/null

real 0m0.015s
user 0m0.009s
sys 0m0.007s

ammon@zapp:~$ time ./tail-array.php a.log >/dev/null

real 0m0.016s
user 0m0.010s
sys 0m0.006s
[/code]

Comparable enough. But given a 50,004 line (3.3mb) log file:

The difference becomes quite clear. However… what if my log file grows obscenely large? I’ve got a 9 million line log file (1.6gb) lying around to test with…

The file() method crashes because it can’t allocate enough ram to hold a 9 million element array and the array method takes almost 20 seconds to execute. It’s slow… but at least it works.

Of course, there are other methods. The one I finally settled on is this:

This method doesn’t waste time reading the bulk of the file. It jumps to the end and scans backward until enough newlines have been located. The only problem here is that your average filesystem isn’t optimized for reading backwards… but since we’re not really reading very much data, it doesn’t much matter.

Performance is a trifle slower on small files, but it’s astronomically better on long ones. This is similar to the method used by most unix ‘tail’ commands, and is the clear winner for actual use in my application.

Of course, it needs a bit of cleanup from the state I’ve provided it in, and isn’t appropriate for all environments… but it’s a trifle better than requiring 20 seconds and 20gb of ram to execute 😉

flash policy service daemon

Sorry it took me so long to post this, but WordPress 2.5 doesn’t seem to like me trying to upload gz/zip files, so I had to upload the source manually.

Well, it’s been months since I promised to post some usable socket policy service code, so I will.

The script here is meant to serve as a good starting point for people whose servers need to allow flash clients to make socket connections. I have not actually used this exact code in a production environment, but I have been using code that is 99% identical for a while now. I am confident that any blatant flaws are the result of simple copy-paste errors as I compiled the package. Please let me know if you find any.

I have however, stress tested the heck out of this service. One instance successfully served up over 16000 policy file requests fed into it as rapidly as I could send them. The same networking code has also handled requests from at least 100 different hosts at roughly the same time.

Everything has been combined into a single cli php script that requires no special installation. Just plop it down on the server and run it as root. It will take care of the rest. The config defaults should be safe, but you probably want to specify them more clearly – just to be safe.

The daemon is made of three classes:

  • Logger – A rudimentary log file management class that I copy from project to project in one form or another. The included version is stripped down from some of the other versions I’ve written, and I’m planning on releasing a more feature-rich version in the future.
  • Daemon – A simple class for daemonizing a process. Adapted and re-adapted countless times from an original php4 class I found on the net a few years ago by some guy named Seth (whose email domain no longer exists).
  • FlashPolicyService – The meat and potatoes, a child of Daemon. Mostly, this is just the requisite networking code and glue to make everything work together.

As with any of my other code, this is licensed under CC Attribution 3.0.

Download:

Source code after the jump.
Continue reading flash policy service daemon

lazy image browser

The other day, I heard a few people talking about needing an easy way to browse images on a remote Apache server that has Indexes disabled.

They had a ~20 line php script that they were dropping into each directory in order to generate indices. The problem came when they started organizing the images into subdirectories. Eventually, it became necessary to copy the new script into a mind-bogglingly large number of directories. Inevitably, dirs were missed, etc…

I interjected that I could probably fix their problem in 30 minutes.

So I did.

It’s not elegant. It’s not pretty. It has plenty of room for improvement – it’ll generate links to Windows explorer thumbnail db’s, etc… But it is fast and should be moderately secure. Just drop it in the root directory of your image structure and you’re good.

new flash security policies

So… I am not happy with Adobe right now. With the push of Flash Player 9,0,115,0 “moviestar”, which included such awesome features as H.264 and AAC codec support and improvements to fullscreen mode, they kind of ambushed me with some sweeping changes to their security policy.

I’d been running pre-release nightly builds of the player since 9,0,60,x… and had noticed some strange warnings. Mysterious “Socket Security Error #2048” exceptions that were being thrown at random – even though I was serving an appropriate (for the time) crossdomain.xml file, unexplained timeouts attempting to talk to an xml socket server when I was very clearly not attempting to do any such thing, etc… My regularly repeated attempts to find documentation on what the warnings actually meant proved fruitless. I believe that is because the appropriate document was not actually released to the public until 9,0,115,0 was released.

Now, the bit where they improved the format for crossdomain.xml files doesn’t really affect me one way or the other. I approve of the improvements but could really care less in this case. They don’t really affect anything I’m doing.

The part that really chaps my hide is the fact that they’ve completely redone the way that socket security policies are handled. The important parts:

  • A SWF file may no longer make a socket connection to its own domain without a socket policy file. Prior to version 9,0,115,0, a SWF file was permitted to make socket connections to ports 1024 or greater in its own domain without a policy file.
  • HTTP policy files may no longer be used to authorize socket connections. Prior to version 9,0,115,0, an HTTP policy file, served from the master location of /crossdomain.xml on port 80, could be used to authorize a socket connection to any port 1024 or greater on the same host.

That’s right. Your socket policy data can’t live in the sitewide crossdomain.xml file that Apache serves any more.

Flash Player 9,0,115,0 introduces a concept of socket master policy files, which are served from the fixed TCP port number 843.

Socket policy files may be obtained from the same port as a main connection (the socket connection being made by ActionScript, which is authorized by a socket policy file), or from a different port, separate from the main connection. If you opt to serve a socket policy file from the same port as a main connection, the server listening on that port must understand socket policy file requests (which are indicated by a transmission of from Flash Player), and must respond differently for policy file requests and main connection requests.

  • When a SWF file attempts to make a socket connection, even to its own domain, Flash Player will first attempt to contact port 843 to see if the host is serving a socket master policy file.

So… regardless of whether you’re even using a custom port 843 client, the Flash Player is going to try to hit it. What if your firewall doesn’t allow/route traffic to sub-1024 ports w/o special configuration? What if you don’t have the access to bind to a sub-1024 port and can’t rewrite your other server process to serve the policy data on its port?

  • Socket meta-policies can only be declared in a socket master policy file. The syntax is the same as for declaring a meta-policy in an URL master policy file, using the <site-control> tag. Socket meta-policies cannot be declared in HTTP response headers, as HTTP is not involved.

This implies that you can’t even tell apache to listen to port 843 and serve up the data. You HAVE to either maintain a separate server process specifically for the purpose of serving this policy data, or you have to edit the process that SWF’s are connecting to and make them serve the data..

As of the time of this writing (10 days after moviestar’s release), they have yet to release promised help on how to deploy a solution to these new changes. Granted, the one article they did release explains what needs to be done in high level terms. It was sufficient to help me out. I wrote a server that simply listens on port 843 and spews the required xml. But… I’d have really appreciated specific examples, and I suspect plenty of people would appreciate drop-in solutions to the issue.

A 5-minute skeleton implementation (not recommended for production use by any means) written as a PHP cli script might look something like this:

I’ll try to make my production version of this a bit more suitable for public consumption and release it as soon as I can.

The random #2048 security errors continue, despite having deployed my port 843 policy xml server. Granted, they happen less than before… but they still happen. And even when my policy server isn’t running, the errors aren’t thrown 100% of the time. This just baffles me. If they were consistent, that would be one thing. But when you get a security error 1 time in 20… that’s not security, that’s not even a lame deterrent. It’s just incentive to hammer the same port over and over again until something finally gives.

Now, I admit that I could be wrong here… but I’ve re-re-read the documentation on these policies a few times now, and cannot find any reason for the behaviors I’m seeing.

update

On April 22nd, 2008, I released a much better, much more reliable version of this daemon. Head over there for more details and source code.

php post method

Yes, I know you can do this via cURL. This was explicitly written for a case when that particular luxury was not available to me.

So… it’s been a while since I’ve written a post, and I’m feeling the withdrawals. I’ve got 16 half-written posts waiting in the queue and two more that I want to write but haven’t even started. It’s also time for another round of anime reviews; I’ve got two video games I want to review…

And I am having to play the “too busy with RL” card.

In stead, I present a function that I wrote this morning. It’s been done countless times before, and you can probably find incredibly similar code out there already, but it’s the first time I’ve ever actually needed something quite like this.

This method could easily be modified to send GET data or to talk over a different port. It could also probably actually do something with the response buffer (check for 200, etc…). It could be a bit more fault tolerant, etc… but that’s not what I need for the application at hand.

For other examples of how people have done similar/related things, take a look at the comments on the fsockopen() documentation page.

ioerror on https under ie7

Last week week, I noticed a strange problem with a project I am working on. The SWF worked fine on XP, Linux, Vista, and OSX. It worked under Firefox, Opera, and Safari. It… loaded under IE7, and then just sort of sat there and pretended that the web services it was trying to call were broken. After poking things for a bit, I sent an email to the Flashcoders mailing list:

I have a swf that is being loaded off of an https server. As it fires up, it attempts to call a simple authentication service that lives on the same host. This works fine under Firefox, Opera, and Safari.

However, under IE, it throws an exception:

[IOErrorEvent type=”ioError” bubbles=false cancelable=false eventPhase=2 text=”Error #2032: Stream Error. URL: https://host/path/script.php?username=allaryin&passwd=hash”]

Obviously, if I just load the url directly into IE, it loads fine. This problem only occurs when flash tries to load the url for me.

When I monitor the query with Charles, it shows that the request is being made correctly and that the information I am expecting is successfully being returned. However, Flash is apparently ignoring the response.

This behavior has been observed on different machines, running both XP and Vista.

Thoughts?

A few days after sending this email, I’d received no response other than an IM from a friend on the list who didn’t know the answer either. So… I resumed consultation of the Google. I dug through ML archives. I read IRC logs. Eventually, I stumbled across mention of a blog post back in 2005 that had addressed a similar problem under IE6. Unfortunately, the site hosting this old blog has ceased to exist/function. So, I found it on the Internet Archive:

http://web.archive.org/web/20070521185428/http://www.gmrweb.net/2005/08/18/flash-remoting-https-internet-explorer/

The post mentions a few potential solutions to the problem such as doing some http header management in Apache. I tried the suggested changes (in the PHP, I didn’t have the access/desire to tweak Apache at the time):

This didn’t have any effect on my particular problem. Charles showed that the requested headers were being sent correctly, but Flash + IE7 + HTTPS still failed to talk to my web service.

So, I poked around the net some more without coming up with any helpful solutions to the problem. Returning to the archived blog post, I read the comments and saw another solution proposed. Someone said that simply sending an empty pragma header seemed to help in cases where IE was having difficulty dealing with PHP sessions over HTTPS. Specifically, the pragma header needed to be flushed after the session had started.

And we were using a session variable…

And it works now.

So… yeah. Hooray for obscure comments on wayback machine archived blog posts addressing a similar problem 😉