We fixed an issue where you couldn't run Exceptionless on AWS because you couldn't install plugins on their ElasticSearch service. #280
The archival format has also been change from projected\year\month\day to year\month\day\hour\projectid. This allows you to quickly restore or download all events in a time period without enumerating all backed up projects.
Watch out this week, Blake's on fire! We're talking email loggings, UI tweaks, Exceptionless.NET updates and fixes, Foundatio updates, and Foundatio.Repositories updates. Lot's going on, let's check it out in this week's Live Code Demo.
Blake's back at it this week, onward to something new after the Exceptionless 4.0 launch. Today we talk deduplicate event totals, setting minimum log levels, updates to Foundatio, and tracking down an event processor job issue!
We had previously reverted to earlier commits that showed the deduplicated event total separately from the event total, but since we are running on new indexes now, we are able to reliably include the deduplicated total as the total number of events. GitHub Issue #270
You can now set the min log level in configuration using:
The Foundatio updates we made include performance and handling improvements to the message bus. Learn more about these improvements on GitHub. We also added IHybridCacheClient interface marker, giving you granular control over dependency injection. Now, you can inject either a cache client or a hybrid cache client much easier. Learn more.
Lastly for this week, we are working on tracking down the root cause of an issue with the event processor job that is causing it to stop processing events. More on that soon!
During the Exceptionless 4.0 Release, we covered a lot of enhancements and changes. In this week's Live Code Demo, Blake reviews the 4.0 release notes and talks through other tweaks and post-launch cleanup.
That's right folks, Exceptionless 4.0 is here and we've got the full low-down on all the improvements, upgrades, and enhancements in this week's blog article, along with how to upgrade instructions for self hosters.
The primary focus of version 4.0 was to add support for Elasticsearch 5. By migrating from Elasticsearch 1.7 to 5, we removed growing technical debt and benefited greatly in the process. Here is an overview of the benefits gained by migrating to Elasticsearch 5 in 4.0.0.
Since we first migrated to Elasticsearch 1.x from MongoDB, we've learned a lot of hard lessons. With 4.0 and the upgrade to Elasticsearch 5, we took all those lessons and applied them to the new release and infrastructure setup. Here are some performance improvements and notes that came out of those implementations.
We've moved from monthly indexes to daily indexes. This change means that when you are filtering by the last 24 hours or last week, you are greatly reducing the amount of work Elasticsearch has to do to return the requested data.
Because of this, the average use case will see between 25% and 80% improvement to indexing throughput. This means that, with the same hardware, we can index the same documents much faster!
We moved from Groovy scripts to Painless scripts, which are four times faster. By moving to Painless scripts, we also greatly simplified the burden of having to modify configuration files to enable scripting!
Elasticsearch has added a lot of new data types since 1.7, and we are now taking advantage of them to reduce memory, storage and query costs. This means we can query the same data faster, using less memory doing it. We've also set up more sensible defaults to ensure we don't index very long strings.
We've also made several performance and reliability improvements to snapshots (backups).
Our goal is to have everyone be able to setup and use Exceptionless in a matter of minutes. One of the areas that had hendered a lot of self hosters in the past was forgetting to update the elasticsearch.yml file, which is no longer an issue with the migration to Painless scripts. If you're a self hoster, check out the
The move to using Painless scripts as part of bulk updates and ingest pipelines reduced the burden of having to modify configuration files to enable scripting! Long gone are the times where you would have to reset your setup because you missed a configuration step.
We've also added various maintenance jobs that handle backups and restores automatically, so you don't have to worry about losing your data!
Future upgrades should be seamless as Elasticsearch now handles reindexing out of the box! Once you have migrated your data to Elasticsearch 5, we think future major upgrades will just be handled by the Exceptionless app itself!
You can now also use Docker or Azure arm templates to quickly set up a cluster. We really like this direction and will continue moving down this path, hopefully getting to the point where you can click a single button and have an Exceptionless production instance running locally!
With Exceptionless 4.0.0, we continued to focus on finishing up backend improvements to both our repositories and parsers that we made in the 3.5 releases. We feel that all the pieces are finally in place to allow us to do custom dashboards in the near future, something we talk about in our 2017 Roadmap blog post.
The only users that need to worry about upgrading anything for this new release are self hosters. If you are self hosting Exceptionless, please review the Self Hosting Documentation, which contains information about upgrading your existing install.
Also, please take a look at the change log for a full list of the changes.
We’re always striving to improve the efficiency of Exceptionless and all of our projects. If you see any room for improvement or have any comments when using anything from us, please send us an in-app message, submit a GitHub issue or contact us on the website.
In this live code review, we go over some of the recent changes and processes to get ready for Exceptionless 4.0, which include implementing a new Elasticsearch cluster, reindexing into it using the Elastic Arm templates, and updating existing events mapping.
Upgrading to Elasticsearch 5 has been in the works for some time now, with early testing starting in late 2016. As of January 23rd, we have officially pushed the upgrade to production and are looking for feedback, missed bugs, and performance improvement reports from anyone using it!
Elasticsearch 5 is the latest version of ElasticSearch and brings in massive improvements to event indexing speed, reduced memory sizes, and much more.
The Elasticsearch libraries support .NET Core, bringing us one step closer to fulfilling a vision that includes cross-platform support.
Our new implementation is built on a new, faster Microsoft Azure and SSD storage infrastructure. This will greatly increase the indexing and searching performance, reducing dashboard latencies, among other things.
With the power of Foundatio.Parsers project, we have gained the ability to consume generic aggregations and pivot/report on data any way our users like it.
This upgrade also makes it easier to self host through the use of containers. Our goal is to move the entire app to use containers, which will allow you to self host in seconds!
We will now be doing daily indexes for more performance, fine grained backups, and the ability to more quickly change our indexes.
We know the dashboard is crucial for many things, and we want to make them more customizable to fit each user's individual cases. At the same time, we want to make sure we maximize business value. So, we plan on working on several features and functional aspects of dashboards, namely:
Creating pre-define dashboards that are more granular, such as device usage, breakdown by exception type, etc.
Allow users to create their own custom dashboards.
We really want to add new native clients to Exceptionless, and are including that as a major goal for 2017. Everything we are working on is moving in that direction, one way or the other, and we hope to make further announcements in this department soon!
We are, of course, always accepting pull requests as well!