Kusto appender for Log4j.
MIT License
Apache Log4J 2 sink for Azure Data Explorer.
This sink allows you to stream your log data to Azure Data Explorer, Azure Synapse Data Explorer, and Real time analytics in Fabric.
With interactive login, application developers can use Kusto Free to debug and log data from their applications without having to provision a cluster. Set the parameter useInteractiveAuth to true (and tenant if applicable) to use interactive login.
Log4j2 is widely used as logging tool. Kusto implementation is used in conjunction with RollingFileAppender with KustoStrategy. The key reason for using a strategy is to have redundancy in storage of logs and re-transmit the log files.
To provide data transmission redundancy, the rolled over log files are transmitted to Kusto. Transmission of the files are attempted 3 times with a configured time window
The key parameters for rolling file are as documented in the Rolling file log4j configuration
Configurations for using the Kusto log4j appender is as follows
KustoStrategy
To attempt retries in case of ingestion failures, retransmission is attempted with the following configuration. 3 retries are attempted to ingest the logs. In the event that the file cannot be ingested it gets moved to the backout directory in the same path defined in fileName
<Configuration status="WARN">
<Appenders>
<RollingFile name="ADXRollingFile" fileName="<fileName>"
filePattern="<filePattern>">
<KustoStrategy
clusterIngestUrl="${env:LOG4J2_ADX_INGEST_CLUSTER_URL}"
appId="${env:LOG4J2_ADX_APP_ID}"
appKey="${env:LOG4J2_ADX_APP_KEY}"
appTenant="${env:LOG4J2_ADX_TENANT_ID}"
dbName="${env:LOG4J2_ADX_DB_NAME}"
tableName=""
logTableMapping=""
mappingType=""
flushImmediately=""
proxyUrl=""
backOffMinMinutes=""
backOffMaxMinutes=""
/>
<CsvLogEventLayout delimiter="," quoteMode="ALL"/>
<!-- References policies from https://logging.apache.org/log4j/2.x/manual/appenders.html -->
<Policies>
<!-- Recommended size is 4 MB -->
<SizeBasedTriggeringPolicy size="4 MB"/>
<!--
The interval determines in conjunction with file pattern the time for rollup. If file has pattern
file-yyyy-MM-dd-hh-mm.log then rollover happens evey 5 minutes (interval below)
With a date pattern file-yyyy-MM-dd-hh.log with hours as the most specific item, rollover would happen
every 5 hours
-->
<TimeBasedTriggeringPolicy interval="5" modulate="true"/>
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Root level="debug" additivity="false">
<AppenderRef ref="ADXRollingFile"/>
</Root>
</Loggers>
</Configuration>
In power-shell the following can be set
$env:LOG4J2_ADX_DB_NAME="<db-name>"
$env:LOG4J2_ADX_TENANT_ID="<tenant-id>"
$env:LOG4J2_ADX_INGEST_CLUSTER_URL="https://ingest-<cluster>.kusto.windows.net"
$env:LOG4J2_ADX_APP_ID="<app-id>"
$env:LOG4J2_ADX_APP_KEY="<app-key>"
followed by
mvn clean compiler:compile compiler:testCompile surefire:test
If you are a maven user, maven dependency plugin can resolve the dependencies. Note : The library uses resilience4j which is brought in automatically by the dependencies to perform retries.
To use the library in an application, add the following dependency in maven
<dependency>
<groupId>com.microsoft.azure.kusto</groupId>
<artifactId>azure-kusto-log4j</artifactId>
<version>1.0.0</version>
</dependency>
The library expects that log4j-core
is provided as a dependency in the application. This needs to be included as a
dependency, this provides flexibility in using a custom log4j version core library in the application.
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j.version}</version>
</dependency>