trashlog(TL)

subj is a log collecting service for large microservice systems. It provides an grpc inteface for communicating with log emitters. I've choose grpc because it pretty nice, does't need text data overhead due to data transporting and most of backend languages has own protoc to generate clients using provided .proto file. For statistics database TL use clickhouse as one of the best OLAP DB. On the other hand was Apache Cassandra but ClickHouse a way more flexible. If ClickHouse is down TL store unprocessed records to local bbolt storage. Clickhouse has some constrains on inserting such as it may crush on multiple discrete inserts, so I implement batch inserting every second

Potential troubles

• with large amount of incoming data batch may retard. I mean that there are may be more than one second to batch iteration. I have implementation of zero allocation workerpool and I can use it there but it is static workerpool without growing and it may create the same bottleneck but wider(n times wider, where n is workerpool size). On the other hand I can create goroutine for each batch to avoid retarding, but it consume more RAM. The worst of is is that the more retarding batch is than larger next batch

Requirements

• docker.io
• ClickHouse server version 22.1.3.7 (official build).

you can either install them manually or use make install command

HOW TO USE

grpc client generation

just get protoc compiler for your language, go to the root of project and... protoc -I ./proto ./proto/trashlog.proto --<YOUR LOVEABLE LANGUAGE>_out=plugins=grpc:<PATH TO STORE GENERATED DATA>

DISCLAIMER: clickhouse table has strict structure of fields so I don't recommend to try to store fields that are absent.

ADDING FIELDS

to add fields use GetFields method for obtain current fields with their types and Modyfy method for add fields. Thank Gooooooood for proto files, they are pretty impressive and I don't need to describe structure of every request. I just need describe the right way to use them.

First of all you should know that there are no convenient way to remove or to modify table field in clickhouse. if you'll try to make it automatically you will have chance to broke table, reach deadlock or stuck table for hours. so for avoid this there are no request for removing or modifying table. if you sure that you need to do it - do it youself in clickhouse-client

So if you want to add field you should first GetFields to be sure that field with such name and type doesn't exists. Than you can fire Modify request. there are list of acceptable types for fields

			"String",
			"Int64",
			"Float64",
			"UInt8",

Storing logrecords

I think that it is bad idea to send an unary request for every logrecord to store when we can use streams isn't it? so just create one Valve stream and send data in it every time when you need to store log event

WRAPPERS

TL provide some zap.LoggerCore wrappers to make using of trashlog more convinient and simple. There are two ways of logging - to telegram channel by BotAPI or to trashlog server. Strongly recommended use github.com/themakers/hlog as log formatter

ZAPTG

to use it just create a new zaptg core and wrap old logger with new core. ctx - current app context zap.InfoLevel - min loglevel to store - token priveded by BotFather version.Release - version of app. should use git tag version.Commit - current commit 0 - build time

tel, err := zaptg.NewCore(
	ctx,
	zap.InfoLevel,
	"<bot token>",
	version.Release,
	version.Commit,
	0,
	<channel id>,
)

if err != nil {
	panic(err)
}
logger = logger.WithOptions(zap.WrapCore(func(core zapcore.Core) zapcore.Core {
	return zapcore.NewTee(core, tel)
}))

ZAPTRASHLOG

Wrapper to send data to trashlog server to use it just create a new zaptg core and wrap old logger with new core. ctx - current app context zap.InfoLevel - min loglevel to store - token priveded by BotFather v.Release - version of app. should use git tag v.Commit - current commit time.Now().Unix() - run time

tl, err := zaptrashlog.NewCore(
	ctx,
	zapcore.DebugLevel,
	"<trashlog uri>",
	v.Release,
	v.Commit,
	time.Now().Unix(),
)

if err != nil {
	panic(err)
}
logger = logger.WithOptions(zap.WrapCore(func(core zapcore.Core) zapcore.Core {
	return zapcore.NewTee(core, tel)
}))