Tips and tricks 2

Continuing about communication errors...

AS I said, there are some types of errors: timeout, packet splitting and buffer overflow. Some methods used to evade timeout errors are usefull and for solving packet splitting errors.

If you send some type of data in all piece, not always data comes to end point in one piece. Often occurs some delay in middle of packet. Packet received often means, some data has been gotten, but not necessary all data. Packet start and end characters, whereas  data packet length is helpful for determining where is packet end.

Packet splitting errors only occur in receiving end. Simple and effective algorithm of data receiving and processing can be used in receiving part, as follows:

1. For every new connection, create new free buffer, if it is a reconnection, use old buffer.
2. When data is received, all data should be added to the end off additional buffer.
3. Constantly check buffer, if it contains correct data. It's good to check for start, end characters, length and integrity.
4. Remove and precess data, passed step 3 from start of the buffer.

For single communication, it's simple to buffer data. In multicommunication, there are some challenges to distribute data in buffer. But, there are methods and for that, like network packet always has sender and receiver ip and mac addresses, and that information can be used to distribute data in buffer.

Tips and tricks 1

There are special chapter in hardware programing: hardware interaction. Although there are many types of interaction, like TPC/IP, USB, Serial, Bluetooth and etc, some main communication problems are in common. And those problems come from main types of timeout, packet splitting and buffer overflow.

Easiest to correct of those are timeout errors. But for finding, there are no easiest and hardiest... So, for correcting error, it's needed to find it. Communication errors has a big habbit, those wont show up in step by step mode or highly controlled enviroment. Best practice - to log everything at every level, starting lowest level as can be done. If it can be done, connect packet sniffer in the line, or some device with capabilities of logging and retransmitting packet.

Back to timeout errors. It must be remembered, that not all devices are super fast machines and data to travel through the line also cunsumes time. And all devices on the line consumes time... Serial to USB converter, TCP/IP hub, router and etc...

For evading timeout errors, further corrections are needed:

Correct timeout calculation: time_for_transmit_on_byte * byte_count.
Timeout must be recalculated before packet is marked for transmitting and for every new packet

Timeout timer must be started at packed start character arrival/send, not at comunication initiation

Sending/receiving routine must exit imidiatly after stop character arival/send or transfered bytes count, not at timeout, because of it's a practice for retransmitting packed after some time of not getting packet transmition confirmation.

Succesfull work with timeout is only one side of medal. Sometimes it is hard to determine start, lenght and end of packet... However, it is goog practise to add separate characters for start and end of packet, also, if it's possible, and packet lenght

Http protocol examples

First one - just basic get string...

GET /sample/index.php HTTP/1.1 Host: localhost User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20100101 Firefox/6.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip, deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Connection: keep-alive Cookie: PHPSESSID=7pb4shikglb9snth6dckrqvebsmqceoo    Second one - get string with params   GET /sample/index.php?var1=data1&var2=data2 HTTP/1.1 Host: localhost User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20100101 Firefox/6.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip, deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Connection: keep-alive Cookie: PHPSESSID=7pb4shikglb9snth6dckrqvebsmqceoo  Fird one - post method using application/x-www-form-urlencoded POST /sample/index.php HTTP/1.1 Host: localhost User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20100101 Firefox/6.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip, deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Connection: keep-alive Cookie: test=1; PHPSESSID=7pb4shikglb9snth6dckrqvebsmqceoo Content-Type: application/x-www-form-urlencoded Content-Length: 29 data1=value1&data2=value2 And the last one - multipart/form-data    POST /vietine/filepost.php HTTP/1.1 Host: localhost User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20100101 Firefox/6.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip, deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Connection: keep-alive Referer: http://localhost/vietine/filepost.php Cookie: test=1; PHPSESSID=7pb4shikglb9snth6dckrqvebsmqceoo Content-Type: multipart/form-data; boundary=---------------------------62172895214770983271472579671 Content-Length: 482 -----------------------------62172895214770983271472579671 Content-Disposition: form-data; name="file"; filename="1.txt" Content-Type: text/plain text document data goes here -----------------------------62172895214770983271472579671 Content-Disposition: form-data; name="aditionaldata" datavalue1 -----------------------------62172895214770983271472579671 Content-Disposition: form-data; name="ok" Post -----------------------------62172895214770983271472579671--

Http post formats

So, there are two ways to post data to www server. 

For application/x-www-form-urlencoded, the body of the http  message sent to the server is essentially one giant query string -- name/value pairs are separated by the ampersand (&), and names are separated from values by the equal symbal (=).

Talking about multipart/form-data data is send in pairs too, but each pair is represented as a "part" in a mime message. Parts are separated by a particular string boundary (chosen specifically so that this boundary string does not occur in any of the "value" payloads). The value piece of each name/value pair is now the payload of each part of the mime message.

MIME message saves space, when dealing with large amounts with data, and first one - for short data.

Http explained 3

As I mentioned before, only way to send some data to server using method get - is params, concatenated with file address. And there is limitation in data size. That's a 2K characters. Sometimes that's not enough...
For uploading data bigger, it's used method post. There is two ways of using method post: application/x-www-form-urlencoded and multipart/form-data. Those two ways share same basic principles.  Message is made up form file address, headers followed by posted data.

Http protocol explained 2

Continuing about http protocols

So what happens, when You open telnet, connect to server and type "GET smth?" You basically used get method. That's just simple method for retrieving some file from www server. In addition, there are some space for aditional data. And it comes with headers and get query. Data in headers consist of connection type, data encryption, your browser type, cookies and etc. Data in query stands for file, you want to retrieve, additional parameters and http protocol version numbering.

Server responses to requests always are same - request status, headers and data.

Features, pre-Alpha

Pre-Alpha wwwPoster features:
  - Lightweight console client
  - Can upload large files
  - Supports authentication via cookies