In this live code review, we go over some of the recent changes and processes to get ready for Exceptionless 4.0, which include implementing a new Elasticsearch cluster, reindexing into it using the Elastic Arm templates, and updating existing events mapping.
The Elastic Arm templates automate the setup of an Elasticsearch with a few clicks, which is awesome.
Can you believe it's almost February already! Us either. No rest for the whicked, though, as we sit down to get cranking on our list of Exceptionless changes for 2017.
We have a a great list of features, functionality changes, and enhancements planned, and no doubt we'll work in a few other popular feature requests and, naturally, bug fixes along the way.
Let's take a look at the primary roadmap, shall we?
Upgrading to Elasticsearch 5 has been in the works for some time now, with early testing starting in late 2016. As of January 23rd, we have officially pushed the upgrade to production and are looking for feedback, missed bugs, and performance improvement reports from anyone using it!
Elasticsearch 5 is the latest version of ElasticSearch and brings in massive improvements to event indexing speed, reduced memory sizes, and much more.
The Elasticsearch libraries support .NET Core, bringing us one step closer to fulfilling a vision that includes cross-platform support.
Our new implementation is built on a new, faster Microsoft Azure and SSD storage infrastructure. This will greatly increase the indexing and searching performance, reducing dashboard latencies, among other things.
With the power of Foundatio.Parsers project, we have gained the ability to consume generic aggregations and pivot/report on data any way our users like it.
This upgrade also makes it easier to self host through the use of containers. Our goal is to move the entire app to use containers, which will allow you to self host in seconds!
We will now be doing daily indexes for more performance, fine grained backups, and the ability to more quickly change our indexes.
We want to port the Exceptionless app to .NET Core so we can run it anywhere (macOS, Linux or Windows).
Running on .NET Core also brings massive performance improvements and lower overhead, which will have positive trickle down affects throughout the app.
Once we support .NET Core, we can containerize the entire app, allowing us to scale more easily.
These changes will make self hosting super simple. You'll basically be able to run "docker run -d exceptionless`" and be done!
We know the dashboard is crucial for many things, and we want to make them more customizable to fit each user's individual cases. At the same time, we want to make sure we maximize business value. So, we plan on working on several features and functional aspects of dashboards, namely:
Creating pre-define dashboards that are more granular, such as device usage, breakdown by exception type, etc.
Allow users to create their own custom dashboards.
We really want to add new native clients to Exceptionless, and are including that as a major goal for 2017. Everything we are working on is moving in that direction, one way or the other, and we hope to make further announcements in this department soon!
We are, of course, always accepting pull requests as well!
What have we forgotten? We know everything isn't included on this list, but if we have missed something big that you can't live without, please let us know!
Otherwise, look for a new version announcement very soon, and get ready to upgrade and take advantage of several of these new upcoming changes.
In the first episode our all newweekly live streaming demo session, Blake talks about the recent push to migrate Exceptionless from Elasticsearch 1.x to Elasticsearch 5.x.
We were curious, so we thought we would put together some stats for 2016, along with and a recap of some of the notable changes and progression. Enjoy!
We changed the server licensing to use the Apache license. Now, every Exceptionless project is under the Apache license and there should be no more confusion on how it's licensed. Read more here.
The ability to track and automatically manage a users session was added, giving you visibility into how long and what a user did while they were using your product. This also answers the question, "what did my users do leading up to this exception?" Read more here.
We spent a lot of time this past year making massive performance and reliability improvements across the board. Tons of bugs got fixed along the way, and additions like bulk updates and deletes, along with optimized query generation got squeezed in there, too! One big factor here is that it has allowed us to move to Elasticsearch 5 much quicker because all of these improvements allowed us to do things more generically throughout. Much less code to update/change!
Elasticsearch 5
Like we mentioned above, we've done a ton of work to ensure we can move to Elasticsearch 5:
Insanely fast document indexing and searching
Dynamic aggregations (allowing us to do completely customize stats on any data)
Reindexing support
Daily indexes
Making it easier to self host
Getting one step closer to server being able to run cross-platform
We have been doing QA on the Elasticsearch 5 migration since the end of the year and it is currently still in progress - coming soon!
We've got big things coming up, like Exceptionless 5.0 with Elasticsearch 5, which should boost performance gains like 80% faster indexing, lower memory usage, and native reindexing support! No slowing down over here, so stay tuned and enjoy the ride.
And, as always, if you have anything you'd like to see in upcoming Exceptionless releases, please let us know!
Today we've got a great example of Exceptionless helping software developers identify major issues in their web applications affecting thousands of users.
In this case, the problem was major enough that the development team stopped using WPF and rewrote their entire UI layer!
Here's what Will Graham, CEVO, Inc. developer, had to say when we asked him a few Exceptionless case study questions.
"The issue was so prevalent we completely ditched WPF and rewrote our UI layer in GDI+/WinForms" - Will Graham, CEVO, Inc
Will:
Definitely crashes related to .NET Framework installation corruptions on end-user machines. While Exceptionless didn't (and can't) immediately point and say, "here's how to fix it", it did shine light on an issue that was affecting approximately 5% of our 500,000 installed base. The issue was so prevalent we completely ditched WPF and rewrote our UI layer in GDI+/WinForms, but Exceptionless gave us insight to see the problem and how many users were being affected.
"Exceptionless has changed our internal development process and how we approach code. Proper error handling and visibility is now a first-class priority for us and Exceptionless makes it super easy." - Will Graham, CEVO, Inc
Will:
Yes and no. Error handling and tracking had always been more of an afterthought - like, something is wrong, we need to look into it now. We weren't aware of issues until after our users reported it and filtering out the noise to find the signal of real issues was a time consuming process. Running on the Azure stack, we were frustrated with the information provided in the default diagnostics logging.
When we added Exceptionless, we knew we had problems with our system, we just didn't know where and to what extent. To be honest, Exceptionless has changed our internal development process and how we approach code. Proper error handling and visibility is now a first-class priority for us and Exceptionless makes it super easy.
The biggest bug would have to be intermittent failures on our Service Bus instance. Using the metrics provided by Exceptionless, we were able to spend time implementing fault tolerance in very focused areas of the code based on incidence rate and end-user impact.
I'm not a huge fan of the current dashboard. It's effective, but with the amount of data Exceptionless has at it's disposal, I feel there's more interesting (and actionable) information that can be shown.
Thanks for your time, Will! We love to hear from our users, and we'll definitely take that dashboard feedback and see what we can do there to make it more useful.
Do you have feedback on Exceptionless? Has it helped you find bugs and beat them into submission? Join us in the Exceptionless Discord and let us know!
(/assets/versioning.png)Do you get annoyed and overwhelmed by event and error notifications?
You probably have more than one version of your application running, and often older versions of your app may still be triggering events that have been fixed in newer versions.
**Those events and notifications aren't very **helpful, so we implemented a versioning system that allows you to set an application version on all events!
After setting an application version, when you mark an event fixed and give it a version number, it will only regress if it occurs again in a newer version of your app.
That means there is less noise for you to wade through, and you can focus on new issues in your application without seeing old or non-relevant events constantly.
Setting an application version in Exceptionless is easy. By default, we attempt to resolve one automatically based on assembly attributes, but we recommend specifying one yourself for improved reliability and accuracy using the following examples.
Great! Now, when you mark an error stack as fixed and enter the version that you fixed it in, that event stack will have a Fixed In [Version] tag and will only regress if it occurs again in a later version of your app. If it does regress, the stack then gets the REGRESSED tag.
(/assets/regressed.jpg)
Regressed
If you would like to view fixed events, you can always use the * wildcard or fixed:true in search.
We hope you find this feature useful, and as always don't hesitate to leave us feedback over on GitHub or by commenting below.
Have you ever needed to cut through the noise and just focus on one type of event(in real time), such as only error logs, to track down a bug?
Do you want to limit certain types of event reporting(in real time) to save your event quota and limit clutter?
What about controlling your application's settings or featuresin real time via Exceptionless, without having to update your files and deploy your app!?
Well, with our client configuration settings, you can do all that, and more, in real time, on a per-project basis via your Exceptionless dashboard!
Let's say you're on the small plan, and you've got a bunch of warning log events clogging your system that you know about and are working on, but they are pushing you over your plan limits.
#BOOM - set a minimum log level of error, and Exceptionless won't report those warnings anymore and they won't count against your plan limits! This is a great way to get the most out of your Exceptionless plan.
Maybe you're having major issues with an authentication bug, but you've already set minimum log levels to only include errors. Well, now you also want to see the trace values for those events without opening the flood gates for every event by removing your minimum log level.
#NOPROBLEM - Just add a key for authentication that just lets trace events through!
Yup! Our client configurations are basically just a key value pair dictionary, but what makes them powerful and helps them control your application's features is that they get updated in nearly real-time, meaning you can build settings, features, etc into your app that react to value changes, and if you change that configuration setting in Exceptionless, your app will react almost instantly!
This can be super useful, especially if changing your app's settings would normally require you to deploy to production. No need - just use Exceptionless!
First, settings updates do not count towards plan limits. We say that because each time an event is submitted, we send a response header with the current configuration version to the client, and if a newer version is available, it is retrieved and the latest configuration is applied. That means that config changes are nearly real time.
When the client is idle, we also check for config changes, including five seconds after client startup if no events are submitted at startup, and every two minutes after the last event submission.
If the version hasn't changed, nothing is retrieved, limiting data transfer, and no user information is ever sent when checking.
If you do not want the configuration settings to update when idle, you can turn off automatic updates. To do so, please follow the respective .NET or JavaScript/Node.js documentation.
Exceptionless client configurations are a dictionary of key value pairs that can be used to control the behavior of your app in real time by doing things like controlling data exclusions, protecting sensitive data, enabling and disable features, or disabling certain types of events (error, usage, log, 404, or session).
We also have some built in configuration key naming conventions (@@EVENT_TYPE:SOURCE) that the clients recognize for ignoring events based on event type and event source. Just replace EVENT_TYPE part with the event type (E.G., error, log...) and the SOURCE (E.G., exception type or log source) you'd like the setting to apply to. Next, specify key value of false to discard matching events client side. It's worth noting that log event types can also accept a log level value (E.G., Trace, Debug, Info, Warn, Error, or Fatal).
For example, we can use it to turn off all error events of type, lets say, System.ArgumentNullException, by using the key @@error:System.ArgumentNullException and the value false.
Or, we could turn off all error events entirely by using the * wildcard. So, the key would be @@error:* and the value would be false again.
In scenario 1, above, we were trying to save our plan limits by limiting log events to only errors. So our Client Configuration key would simply be @@log:* with value Error.
Scenario 2
Here we already limited our log events to errors, but now we're troubleshooting a specific issue with authentication (let's say we're using the AuthController API), so we want to look at the trace messages coming through. We can override any general minimum log levels that we've defined by setting a level for a specific log source. So, all we would do is add the @@log:*AuthController key with value Trace! Then, when the bug's fixed, turn it off as needed.
Scenario 3
This is the cool one. Here you are wanting to, let's say, pass a value for a setting in your app that turns something on or off without having to re-deploy everything. This is super easy to accomplish all we need to do is create a setting which will control our feature! Let's assume we have have a feature flag to show a welcome screen. We will name this feature flag enableWelcomeScreen and create a new configuration setting respectively with a value of true (You can change this value at any time). These changes will be pushed based on the above "How Project Settings Work" section automatically, all we have to do is check the setting as shown below.
C#
usingExceptionless;// Check the configuration settings for our enableWelcomeScreen feature flag with a default value of false.if(ExceptionlessClient.Default.Configuration.Settings.GetBoolean("enableWelcomeScreen",false)){// Show the welcome screen!}
JavaScript
// Check the configuration settings for our enableWelcomeScreen feature flagif(exceptionless.ExceptionlessClient.default.config.settings['enableWelcomeScreen']===true){// Show the welcome screen!}
If you go to Admin > Projects in Exceptionless, you can choose the project you would like to edit the settings for. Each project can have unique settings.
The default tab is "General," which simply houses the project name and attached organization. Nothing fancy here - pretty self explanatory.
This tab is where you can generate an API key for your project. Again, pretty self explanatory. Hit "New API Key" and one gets generated. For more details on API usage, check out the API Usage documentation on GitHub.
This is where you can set data exclusions, customize error stacking, and build in some spam detection to your project.
Data Exclusions
There are several use cases where you might not want to send some data to your Exceptionless project. This field allows you to enter a comma delimited list of field names that will be removed and not transferred to Exceptionless. The perfect example here is a password field, or other personal and sensitive data.
The * wildcard is supported in this field and can be used at the end (password*), beginning(*password), or on both sides (*password*) of the field name to further customize your data exclusion.
Error Stacking
Control over error stacking is another level of customization project settings allows. You can specify user namespaces or common methods to ignore. More details below.
User Namespaces
Here you can enter a comma delimited list of namespace names that own your application. With those in place, the only methods that will be considered stacking targets are ones inside those namespaces.
Common Methods
If your code has shared utility methods that may generate a bunch of errors, this could be useful. Enter a comma delimited list of common method names that shouldn't be used as stacking targets, and they will be ignored.
Spam Detection
Spam is the worst. So, we added a "Spam Detection" list of common user agents that should be ignored, which you can add to as you see fit. This eliminates a lot of noise, and can be customized to help trim even more depending on your application.
Along with the comma delimited list of user agents to ignore, you can also tick the box that says "Reduce noise by automatically hiding high volumes of events coming from a single client IP address." This can ward off large numbers of events being submitted by a spammer or attack on your app.
Integrations
Integrations with tools like Slack, HipChat, JIRA, Basecamp, and others are very popular and can add a level of automated notifications, etc, to your workflow. So, on the integrations tab of your project's configuration you can create web hooks to integrate with your service or others as mentioned. Each web hook has a URL that it can call, and options for when it should be called. When a selected event occurs, a POST request is submitted with either event or stack data in JSON format. For more details and sample data, visit the Exceptionless integrations documentation.
We hope this will help you make the most out of your Exceptionless projects, allowing you to save some event submissions, get that customization you were looking for, etc.
Please let us know if you have any questions, comments, concerns, bugs, or anything else we can help with!
The latest Exceptionless release has several additions we think most of our users will find helpful. We sat down and worked on the UI, fixed some bugs, and spend a considerable amount of time improving reliability and efficiency of some of the primary pieces of the app.
If you're a self hoster, you'll need to upgrade your existing install, but if you're hosting with us there is no action required on your part to experiences the Exceptionless 3.4.
The new most users dashboard allows you a quick view of events sorted by the highest number of affected users. This is great for helping prioritize your work pipeline.
Also, as an aside, we've added the users affected column to the dashboard. We know some of you guys will find that helpful.
MacOS & Linux keyboard shortcut support has been added, as well as additional shortcuts such as C to chat with support, S to focus the search bar, and ga to go to your account. Hit SHIFT + / (also known as ? ) to access the keyboard shortcut list on any screen.
As an aside here, there is also now a </> button near the top of the event occurrence that lets you quickly copy the JSON to your clipboard with a click.
We made several reliability and performance enhancements to queue and job processing. A few specific examples include fixing auto-abandoned jobs and instances where batch events weren't being requeued.
Previously we had worked on making heartbeat events efficient so we didn't have to count them toward event quotas, and with this release we've added new API Endpoints that allow clients to submit those heartbeats cheaply.
Support has been added for Active Directory Authentication. Thanks @laughinggoose! To enable this feature, head over to the Active Directory Authentication documentation page on GitHub.
This Count property was added to the event model that tracks deduplicated events and allows for some pretty cool metrics from here on out while avoiding the full cost of storing every event.
MaximumRetentionDays is pretty self explanatory. It controls the max retention perdiod for events, which allows the retention job and plans to be smarter about cleaning up old data.
The NuGet package now supports .NET Standard 1.2+, PCL Profile 151, and .NET 4.5.
To make upgrading easy, the Exceptionless.Portable NuGet package still exists and is dependent on the Exceptionless Package.
If .NET 4.0 is a requirement for your projects, you will need to continue using the Exceptionless.NET v3.5 NuGet Packages, which are not being deprecated.
As a matter of housekeeping, we removed the ExceptionlessClient.Current property, which was replaced with ExceptionlessClient.Default in v2.0.0, and the configuration EnableSSL properties (also deprecated in v2).
Updating the NuGet packages should be all you need to do if you are upgrading from Exceptionless.NET v2 or v3. Take a look at the upgrade guide if you have any questions.
That's right! We've re-imagined how session heartbeats and session end events should work on the back end and were able to make them much more efficient, allowing us to stop counting them toward user plan limits!
This blog post explains our original goals and implementation of these session events, and how we were able to retain the same functionality of the feature while limiting resource usage.
Our hope is that this will obviously make our users happy, but also that all the developers out there can benefit from our process and solution.
In the beginning, we set out to create a sessions feature that allowed our users to submit a session start event and the session would be automatically updated by sending a session heartbeat event, as well as a session end event, respectively.
These session heartbeats and session end events were meant to be session markers to show that a user was active or a session had ended.
We wanted to leverage our existing infrastructure, and the easiest way to do so was to introduce new event types that we recognized. This meant that these events went through the client side plugins (extra work) and server side processing. As such, there was no way to tell these events apart from any other event.
And because of that, they counted against user plan limits.
After releasing the sessions feature, it didn't take us long to notice that the heartbeats were noise. However, we knew that users wanted to see what their customers or users were doing while being active throughout the session, so we didn't remove them.
We knew they were counting towards plan limits, causing some users to reach theirs quickly, and we knew they were adding noise, thus limiting the feature's value, but we also wanted to keep the feature alive because of the potential value offered. So, we had to react and make the entire feature more feasible, streamlined, and cheap.
We did some thinking (and coding), trying to determine the best way we could provide end users with a great session tracking feature without over-taxing our system in the process, and we were able to come up with a solution!
So, we created a new GET API endpoint /api/v2/events/session/heartbeat (api source) that takes a session id or user id and a flag if the session is closed. This API endpoint then sets a unique session cache key with the current time.
Our existing CloseInactiveSessionsJob.cs was already periodically polling for open sessions to check for inactive sessions so it could automatically close them after a period of time if no session end event was sent, so we just updated this job to check for the unique session cache keys (source) and get the last time a heartbeat was sent in or see if it was closed. It then takes the appropriate action and updates the session event.
Then we just updated the clients to call this new API endpoint when await client.SubmitSessionHeartbeatAsync("id") or await client.SubmitSessionEndAsync("id") is called.
Our new solution gives us the ability to have clients send us session heartbeat and session end information very efficiently, which lets us provide a great session tracking feature without adding any additional cost to our plans!
That's how we like to roll, and we hope you find value from the feature and our run down of the process we went through to get it to our users.
