03. Sept. 2014 by Marek
The main goal of Retrospective is to provide a fast and seamless experience for a user who wants to quickly analyze log files dispersed over distributed environments. Since Retrospective relieves the user from a complicated setup involving the specification of all the details about the environment, knowledge about the environment is relatively restricted. Whilst working on release 3.3.0, we tried to implement approaches, tricks and algorithms which use the available restricted knowledge and ensure that Retrospective is faster and smarter in fulfilling the needs of the user. The main added values are as follows:
Dynamic Tail – Retrospective dynamically includes new appearing files in the monitoring process
Shell Search Optimization – for remote files, the optimization is entirely performed in shell scripts.
Shell Search Narrow-down – for remote files, the date search is narrowed down to a file part that can potentially contain matching log entries
Deep Wildcards – log files can be specified by a wildcard filter which can refer not only to file names but also to directories
More powerful search criteria – the awaited “Not contains” filter and internal capabilities needed for supporting wildcards and regex filters in the near future.
Dynamic Tailing is the answer to situations in which the user operates in an environment of increased dynamics. When, both local and remote, files come and go in a dynamic manner, then the user is forced to reconfigure his/her searching profiles far too often. No more of this nonsense with Dynamic Tailing! Now Retrospective adapts itself without user intervention. The user simply specifies a wildcard filter covering, for example, several directories in which files can dynamically show and disappear. Then, Retrospective tracks the changes and dynamically monitors new files. If you are afraid that you are monitoring a directory in which logs are rotated and it could cause the whole rotated content to be fetched to Retrospective … just relax. Dynamic Tail is smart enough to distinguish the actual new content from the rotated one. Additionally Dynamic Tail uses the resources more effectively by adapting the amount of threads used for tracking remote files: when a file starts to change frequently, then a dedicated thread fetches the content as quickly as possible, otherwise no dedicated thread is consuming resources (both from Retrospective and remote servers) tracking a file which rarely changes.
Search Optimization was already present in the previous releases; however it was performed within the Retrospective JVM. In the case of remote files this was not really efficient as the content needed for analysis had to first be transferred to the JVM. Therefore, here comes Shell Search Optimization, which performs everything on the server side. Was it easy to implement? Not at all! However, since there is not much about POSIX shell and awk that we do not know, we made it. The optimization is so great, you should really give it a try! Searching remote files with date criteria will be a lot faster now. Moreover, Retrospective 3.3.0 introduces an even smarter option that allows the irrelevant content to be skipped during the search with date criteria. When the option is enabled, Retrospective excludes files from the search process if their modification date is earlier than the specified date criteria. If the file was modified in 2012, then obviously there is no reason to open it, when looking for log entries from 2013.
Do you maybe deal with some large log files that need to be searched with a date criteria matching only a fragment of these files? If the answer is “yes”, then Shell Search Narrow-down will definitely make you happy. The implemented procedure uses effective divide and conquer approach to narrow down searching to a part of the file which could possibly match the date/time filter. The rest of the file is excluded from the search process. The procedure tries to achieve the logarithmic computational complexity [the ever desired O(logn)]. This gives a significant performance boost. For example, if the user has a 300 MB file and he/she specifies a date criteria covering 1 MB of the file, then the narrow-down procedure is able to decrease the duration of searching process from tens of seconds to just a few seconds.
If such a path is provided for a search in Retrospective, then all “*.log” files from every day in August 2014 will be searched. In the release 3.3.0, Retrospective UI supports specification of Deep Wildcards only for the last directory level.
Finally, the filtering capabilities of 3.3.0 are enriched. The user is able to define the “Not Contains” filter that specifies what should not appear in the search/tail output. It works in the same way as the well-known “grep –v” … however, it is much more powerful as the “Not Contains” can be arbitrarily joined with other filters (including other instances of “Not Contains”). Moreover, the internal Retrospective capabilities were reworked towards further enrichment of the filtering capabilities. We promise that useful features such as regex and wildcard filters will be covered in one of the upcoming releases.
You have to admit that Retrospective really looks a lot smarter in its 3.3.0 outfit cheeky. If you have already used the previous versions, just grab 3.3.0 for a faster search and make a lot of your current problems just go away. If you have not touched our product before and are still using the good old tail and grep combinations in 10 different shell windows, give yourself a treat and allow Retrospective 3.3.0 to do all the hard work for you. You then can forget about managing the 10 different shell sessions and start focusing on the actual results of searching and tailing. Now wouldn’t that be nice?