Uploading stats to InfluxDB in the C language

Steps

• https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_reference/
• https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/

• https://docs.influxdata.com/influxdb/v1.7/tools/api/

• There are lots of special cases that are confusing.  For example, tags are strings but must not be quoted but data can be a string and must be quoted by using the double quotation character.
• There is also vast duplication. For example, the tags have to be included with every line. If you have tags for the hostname, the operating system name, the hardware architecture, the server serial number, and the model type then that is a large quantity of data being repeated on every line. 

• Diagram of the components
• The use of sockets in C
• The HTTP API like the POST request
• The InfluxDB Line Protocol
• The example code
• Housekeeping
• Compiling and use
• What next?

---

Diagram of components

From the server called blue a socket is created to the server with IP address 9.137.62.12. C language system calls and libraries are used to gather stats that are sent to the Influx server down the socket that uses HTTP POST requests. Finally, the response is read from the socket to confirm the data arrived and was saved.  The data can be lists on InfluxDB or graphed by Grafana.

---

1. The code to create and connect a socket is not going to be covered in detail as you can find many examples on the internet.
2. Line 6 to line 9 for the include files required for the socket() and connect() functions.
3. Line 44 to line 53 for the actual socket() and connect() code.
4. In this example, the IP address of the InfluxDB server and the standard InfluxDB port of 8086 is used. InfluxDB admin can set up to use a different port so check.
5. If you want to supply the hostname instead, then use the gethostbyname() function to convert the hostname to the IP address.
6. The socket() and connect() functions create a working socket bidirectional pipe file to the InfluxDB engine.

1. The most often used operation of the HTTP API is the GET request, which your web browser uses to "get" all the text and images of a web page.
2. However, here we are sending data to InfluxDB - the reverse action so we are using the POST request.
3. POST comes in two parts, in this order
   1. The Header: Contains the specific operation that we are asking InfluxDB to perform.
   2. The Body: Contains the data that we are sending. This data is InfluxDB Line protocol formatted statistics.
4. The Body is prepared first as we need the final size in bytes to create the Header
5. The Body is simple enough in this case and created in line 61.  Ignore the call to the random() function that is used to generate fake data in this example code.  The Body content is the statistics in Line Protocol format and covered in the next section.
6. The Header is HTTP and InfluxDB HTTP API details and created in line 65.
7. An example of the output of this code like looks like this

     POST /write?db=rand&u=Nigel&p=passw0rd HTTP/1.1  
   Host: influx:8086  Content-Length: 38  <empty line>  

8. The details all of these parts are mandatory to get precisely correct, including the spaces and carriage-return and line-feeds:
   1. The POST is HTTP.
   2. The "/write" and following details are in the InfluxDB HTTP API.  It is the command you want InfluxDB to perform.  The "write" meaning we are writing data to InfluxDB.
   3. The "HTTP/1.1" is HTTP and pointing out using version 1.1 of the HTTP protocol.
   4. The "Host: Influx:8086" is InfluxDB HTTP API. I think is to confirm that the programmer knows what they are doing and have the right port number!
   5. The "Content-Length and number" is the size of the Body that must follow the Header.
   6. The "empty line" tells InfluxDB that the header section is finished.
9. The details of the /write part are:
   1. The db= supplies the InfluxDB database name.
   2. The u= supplies the InfluxDB user name.
   3. The p= supplies the InfluxDB user password.
10. Next, in the code, the Header and Body are written to the socket and sent to InfluxDB in line 69 to line 77.
11. We must read the response from InfluxDB to check it was acceptable by InfluxDB. This checking is performed in lines 79 to line 85. 
    If we don't read the responses, the socket could fill up with data and the socket communication stops.
12. It returns an HTTP message, which includes a code number.
    1. We all know a 404 is the code a broken website. 
    2. In this case, code 204 means "No Content" and a success. I like to think code 204 as an acknowledgement of the data sent and "no comment" that is the data was acceptable to InfluxDB and successfully saved in the database.
    3. Other return codes and failures can have some human readable details about what was wrong with the data.

1. Line 61 generates the Influx Line Protocol statistics data
2. Basic format is: measure,tag name=DATA timestamp 
3. Real example for this program: 
   noise,host=blue random=2078917.053  
4. The "noise" is my measurement name. In more realistic examples you might use: cpu, memory, disks.
5. The "host=blue" is a typical tag for server bases stats. This tag allows the graphing tools to find all the stats from this particular server hostname.
6. If you have multiple tags, then tags are comma-separated.  For example, host=blue,os=AIX 
7. <space> delimiter to the next part
8. The "random=2078917.053" is the statistics. Here random is the stat name and it is a floating point number. For integers use a following "i" like temperature=96i and for string data use double quotation marks like os_version="AIX 7.2 TL4"
9. <space> delimiter to the next part
10. In our Real example, the timestamp is missing.  This syntax means InfluxDB itself generates the timestamp at the time of the data arriving.  This option probably what you want as you then don't have to work out the number in the C code.  An example of a time stamp look like this: "158274990936892131". This large value is the number of nanoseconds since 1 Jan 1970 (commonly called the epoch).  I found that I needed 3 spaces at the end of the line to get InfluxDB to realize there was not timestamp - frankly I gave up trying dozens of combinations to get it to work.

• The program loops in the "for" statement LOOPS time
• And pauses for SECONDS each time.
• pexit() is a tine function to report the problem and stop the program.

  +1  /* This is sample C code as an example */      
+2  /* Example of loading stats data into InfluxDB in its Line Protocol format over a network using HTTP POST */      
+3  #include <stdio.h>      
+4  #include <stdlib.h>      
+5  #include <unistd.h>      
+6  #include <sys/types.h>      
+7  #include <sys/socket.h>      
+8  #include <netinet/in.h>      
+9  #include <arpa/inet.h>     
+10     
+11  /* YOU WILL HAVE TO CHANGE THESE FIVE LINES TO MATCH YOUR INFLUXDB CONFIG */     
+12  #define PORT        8086                   /* Port number as an integer - web server default is 80 */     
+13  #define IP_ADDRESS "9.137.62.12"  /* IP Address as a string */     
+14  #define DATABASE "rand"                /* This is the InfluxDB database name */     
+15  #define USERNAME "Nigel"               /* These are the credentials used to access the database */     
+16  #define PASSWORD "passw0rd"     
+17     
+18  /* client endpoint details for a tag: replace with your hostname or use gethostname() */     
+19  #define HOSTNAME "blue"     
+20     
+21  #define SECONDS 15     
+22  #define LOOPS   100     
+23     
+24  #define BUFSIZE 8196     
+25     
+26  pexit(char * msg)     
+27  {     
+28          perror(msg);     
+29          exit(1);     
+30  }     
+31     
+32  main()     
+33  {     
+34  int i;     
+35  int sockfd;     
+36  int loop;     
+37  int ret;     
+38  char header[BUFSIZE];     
+39  char body[BUFSIZE];     
+40  char result[BUFSIZE];     
+41     
+42  static struct sockaddr_in serv_addr; /* static is zero filled on start up */     
+43     
+44          printf("Connecting socket to %s and port %d\n", IP_ADDRESS, PORT);     
+45          if((sockfd = socket(AF_INET, SOCK_STREAM,0)) <0)     
+46                  pexit("socket() failed");     
+47     
+48      serv_addr.sin_family = AF_INET;     
+49      serv_addr.sin_addr.s_addr = inet_addr(IP_ADDRESS);     
+50      serv_addr.sin_port = htons(PORT);     
+51      if(connect(sockfd, (struct sockaddr *)&serv_addr, sizeof(serv_addr)) <0)     
+52          pexit("connect() failed");     
+53     
+54      for(loop=0;i<LOOPS; i++) {     
+55          /* InfluxDB line protocol note:     
+56              measurement name is "noise"     
+57              tag is host=blue - multiple tags separate with comma     
+58              data is random=<number>     
+59              ending epoch time missing (3 spaces) so InfluxDB generates the timestamp */     
+60          /* InfluxDB line protocol note: ending epoch time missing so InfluxDB greates it */     
+61          sprintf(body, "noise,host=%s random=%.3f   \n", HOSTNAME, ((double)(random())/1000.0));     
+62     
+63          /* Note spaces are important and the carriage-returns & newlines */     
+64          /* db= is the datbase name, u= the username and p= the password */     
+65          sprintf(header,     
+66              "POST /write?db=%s&u=%s&p=%s HTTP/1.1\r\nHost: influx:8086\r\nContent-Length: %ld\r\n\r\n",     
+67               DATABASE, USERNAME, PASSWORD, strlen(body));     
+68     
+69          printf("Send to InfluxDB the POST request bytes=%d \n->|%s|<-\n",strlen(header), header);     
+70          write(sockfd, header, strlen(header));
+71          if (ret < 0)     
+72              pexit("Write Header request to InfluxDB failed");     
+73     
+74          printf("Send to InfluxDB the data bytes=%d \n->|%s|<-\n",strlen(body), body);     
+75          ret = write(sockfd, body, strlen(body));     
+76          if (ret < 0)     
+77              pexit("Write Data Body to InfluxDB failed");     
+78     
+79          /* Get back the acknwledgement from InfluxDB */     
+80          /* It worked if you get "HTTP/1.1 204 No Content" and some other fluff */     
+81          ret = read(sockfd, result, sizeof(result));     
+82          if (ret < 0)     
+83              pexit("Reading the result from InfluxDB failed");     
+84          result[ret] = 0; /* terminate string */     
+85          printf("Result returned from InfluxDB. Note:204 is Sucess\n->|%s|<-\n",result);     
+86     
+87          printf(" - - - sleeping for %d secs\n",SECONDS);     
+88          sleep(SECONDS);     +89      }     +90      close(sockfd);     
+91  }

---

Compiling and Use

This code was developed and tested on AIX 7.2 (proper UNIX) and Red Hat RHEL 7.6 (Linux) on POWER9.   

To compile on other Linux variations and MacOS that some small changes might be required.

To compile the program:

  $ cc InfluxDB_C_upload_data_example.c

 Note: you could choose a better program file name (-o add_data) and optimize the program (-O3).

You need to create an InfluxDB database (assuming the name is "rand"):

  $ influx  Connected to http://localhost:8086 version 1.7.7  
InfluxDB shell version: 1.7.7  
> create database rand  
> show databases  name: databases  name  ----  _internal  nextract  njmon  temperature  nimon  rand  
> exit    $ 

Here is a sample of output of the program running correctly (note the default executable name is a.out):

 

  $ ./a.out  
Connecting socket to 9.137.62.12 and port 8086  
Send to InfluxDB the POST request bytes=89   
->|POST /write?db=rand&u=Nigel&p=passw0rd HTTP/1.1  Host: influx:8086  Content-Length: 38  |<-  
Send to InfluxDB the data bytes=38   
->|noise,host=blue random=2078917.053     |<-  
Result returned from InfluxDB. Note:204 is Success  
->|HTTP/1.1 204 No Content  Content-Type: application/json  
Request-Id: b980c0bc-598d-11ea-ab63-3a71321b9604  
X-Influxdb-Build: OSS  
X-Influxdb-Version: 1.7.7  
X-Request-Id: b980c0bc-598d-11ea-ab63-3a71321b9604  
Date: Thu, 27 Feb 2020 18:19:42 GMT  |<-

   - - - sleeping for 15 secs  . . .

To check for data, use the flux command on the InfluxDB server and type the following:

  $ influx  
Connected to http://localhost:8086 version 1.7.7  
InfluxDB shell version: 1.7.7  
> use rand  Using database rand  
> select * from noise  
name: noise  
time                host random  
----                ---- ------  
1582747719067711048 blue 2078917.053  
1582747734118884840 blue 143302.914  
1582747749168937093 blue 1027100.827  
1582747764218897625 blue 1953210.302  
1582748230427490131 blue 2078917.053  
1582748266104203943 blue 2078917.053  
1582748885619743186 blue 2078917.053  
1582748900668948228 blue 143302.914  
1582748915718869526 blue 1027100.827  . . .  
> exit    $  

Here is an example with the wrong database name:

  $ ./a.out                              
Connecting socket to 9.137.62.12 and port 8086  Send to InfluxDB the POST request bytes=95   
->|POST /write?db=XXXXXXX&u=Nigel&p=passw0rd HTTP/1.1  
Host: influx:8086  Content-Length: 38    |<-  Send to InfluxDB the data bytes=38   
->|noise,host=blue random=2078917.053     |<-  Result returned 

from InfluxDB. Note:204 is Success  
->|HTTP/1.1 404 Not Found  Content-Type: application/json  
Request-Id: 6e22ac88-598e-11ea-abc0-3a71321b9604  
X-Influxdb-Build: OSS  
X-Influxdb-Error: database not found: "XXXXXXX"  
X-Influxdb-Version: 1.7.7  
X-Request-Id: 6e22ac88-598e-11ea-abc0-3a71321b9604  
Date: Thu, 27 Feb 2020 18:24:45 GMT  
Content-Length: 44    {"error":"database not found: \"XXXXXXX\""}  |<-   
- - - sleeping for 15 secs  

Nice error reporting:

• 204 = Success - we got the message OK
• 404 = but can't find that database name
• The "database not found: "XXXXXXX" - is a useful message

If the error is in the HTTP parts or formatting of the request, you might get a 403 or 404 error code. 
This result means "the request is not understood, better luck next time!"

---

Graphing that data with Grafana

In the Grafana tool, I added a data source for the rand database and graphs the measurement noise and value=random.

Then, found the data in the timeline and made a few adjustments to colour, set null-value to connected and title.

The Grafana settings are in the following diagram:

---

---

What next?

Multiple measurements

You can send multiple measurements in one go.  Have each record end with three spaces and a newline character in the "body" character array.

noise,host=blue random=2078917.053  

noise,host=blue random=18573.852  

The Content=Length needs to cover all these characters, spaces, and newlines.

Developed a simple C library to hid all these complications

To give you the idea of how library might be implemented.  

Once only:

• set_influx_details(ip_address, port, database, user, passwd)
• set_tags(tags)

Then, for each set measurement:

• One call to set_measure(name)

Then, a series of these functions to add the statistics names and data:

• set_float(name, float)
• set_long(name, long)
• set_string(name, "string")

Ending with a call of push(), which sends the data to InfluxDB.

Limitation

Some InfluxDB installations make use of SSL to encrypt communication - this secure transport layer would be a tricky feature to implement.

- - - The E n d - - -