Take 40% off Logging in Action by entering fccwilkins into the discount code box at checkout at manning.com.

Hello World scenario

The hello world scenario is very simple. We will make use of the fact that Fluentd can receive log events through HTTP and simply see the console record the events. To start with we will push the HTTP events using Postman. The next step will be to extend this slightly to send log events using the LogSimulator.

Hello World configuration

The configuration file used in this example is stripped to the bare minimum, defining just the input and output parameters for each plugin, and a couple of illustrative comments, as shown:

 
 # Hello World configuration will take events received on port 18080 using
 # HTTP as a protocol
  
 # set Fluentd's configuration parameters
 <system>
     Log_Level info #E
 </system>
  
 # define the HTTP source which will provide log events
 <source>    #A
     @type http    #B
     port 18080    #C
 </source>
  
 # accept all log events regardless of tag and write them to the console
 <match *>    #D
     @type stdout
 </match>

#A This is a source directive

#B @type indicates the plugin type

#C lines following a plugin define configuration parameters for that plugin

#D The match directive defines which log events will be allowed to the plugin

#E Set the default log level for Fluentd – as we have set the level to be info, this isn’t strictly necessary as that is the default

Before we run the example, let us take a quick look at the configuration file. As you can see, we have provided some comments within the configuration file. We can comment anywhere within the file using a # character. Then we have used a source directive, to define the origins of log events. In this case, using the built-in HTTP plugin capability that the @type identifies. The following name-value pairs are then treated as properties for that plugin. For example, here we have defined the use of port 18080 to receive log events.

We then define an output using the match directive. The asterisk in the match directive is a wild card, telling the match directive any tag can be processed by the output plugin, in this case, standard out which will appear in the console.

Starting Fluentd

As the Fluentd service is in our PATH we can launch the process with the command fluentd anywhere.  However, without a parameter defining the config location the tool will look in different places depending upon the environment and installation process. For Windows and Linux Fluentd will try to resolve the location /etc/fluent/fluent.conf. As we’re not using the default to start Fluentd, we need to navigate the shell to wherever you have downloaded the configuration file or include the full path to the configuration file as the parameter. Then run the command:

 
 fluentd -c HelloWorld.conf
  

This will startup with Fluentd, and we’ll see the information displayed on the console as things startup, including the configuration being loaded and checked. When running Fluentd or Fluent Bit on Windows, depending upon the permissions for your user account, you may get a prompt as shown in Figure 1. This prompt occurs as Fluentd and Fluent Bit will by default expose access points to the network.

Figure 1. Windows prompting to allow Fluentd or Fluent Bit (depending on what is being started) prompt to approval to use the network

We should, of course, Allow access, without it both Fluentd and Fluent Bit will fail. Within a Linux environment the equivalent controls are established through IPTables rules and possible SELinux configuration (As Linux environments can vary more than Windows, it is worth having a good Linux reference to help understand the setup and troubleshoot any restrictions—Manning has several such titles such as Linux In Motion.

The next step is to send a log event using Postman. Once Postman has started, we need to configure it to send a simple JSON payload to Fluentd. The next screenshot (Figure 2) shows the settings highlighted:

Figure 2. Defined JSON payload to send to Fluentd using Postman

Click on the Send button in Postman, and we’ll see the following result:

Figure 3. Fluentd output after sending the REST event

You may have noticed that in the REST API invocation we have not defined time for the log event, therefore the Fluentd instance will apply the current time.

Whilst this is configuration is as good as ‘chocolate teapot’ as the expression goes. It does illustrate the basic idea of Fluentd. The ability to take log events and direct them (explicitly or implicitly) to an output. Let’s finish this illustration by using the Log Simulator to create a stream of log events.

To run the LogSimulator a new shell is required, and you need to navigate to where the configurations have been downloaded. Within each of the Chapter folders is a folder called SimulatorConfig. Depending upon the chapter you will find one or more properties files. If you look inside the property file, you’ll find a series of key-value pairs that will control the LogSimulator’s behavior. This includes referencing the log file to replay or test data. These references are relative, which does mean we need to be in the correct folder to start the simulator for things to startup correctly. Which happens to be the parent folder to the chapters. We can then start the LogSimulator with the command:

 
 groovy LogSimulator.groovy Chapter2\SimulatorConfig\HelloWorld.properties
  

or if you choose to use the jar file:

 
 java -jar LogSimulator.jar Chapter2\SimulatorConfig\HelloWorld.properties
  

That’s all for this article. If you want to learn more about the book, check it out on our browser-based liveBook platform here.