Live Code Demo - New Query Validation, DI Issues, and Error Handling

This week's live code demo comes to you during a week of final testing and tweaks prior to our launch of Exceptionless 4.0.

Blake dives into and shows you the code for several important improvements and fixes, which we've listed below.

  • We've added a much improved Query Validator to the repositories and Exceptionless.

  • Some DI issues with running jobs under the debugger have also been fixed.

  • Then, we added error handling around heartbeat submission end points.

  • And finally, an issue where the app would blow up when trying to save invalid reference ids has been fixed.

Watch Video

Watch more Live Code Demos

Exceptionless 4.0 Enhancement Overview - Live Code Demo

Exceptionless 4 Enhancement Overview

We recently announced Exceptionless 4.0, but we also recorded a short demo of some of the enhancements and talked through them.

In this live code/feature/functionality review video, Blake talks about:

  • Deploying Elasticsearch 5.x to production with completely new Azure resources.

  • Fixing bugs and doing extensive testing on the new Elasticsearch 5 and Azure implementation.

  • Massive performance boosts due to daily indexes, SSD-backed storage, and bug fixes. These boosts should result in near0instant page loads and huge improvements to session dashboards.

  • Automating cluster setup using Azure CLI and Elasticsearch ARM Templates.

  • A new custom SSH script that allows us to quickly and securely connect to machines, as well as forward ports for things like Kibana.

  • New Azure resources and dashboards.

  • Setting up WebJobs to run as cron jobs.

  • We also found PM2, a new way to manage Node.js services for our StatsD service.

WATCH VIDEO

Watch more Live Code Demos

Exceptionless 4.0 is Here!

Exceptionless 4.0 AnnouncementThat's right folks, Exceptionless 4.0 is here and we've got the full low-down on all the improvements, upgrades, and enhancements in this week's blog article, along with how to upgrade instructions for self hosters.

The primary focus of version 4.0 was to add support for Elasticsearch 5. By migrating from Elasticsearch 1.7 to 5, we removed growing technical debt and benefited greatly in the process. Here is an overview of the benefits gained by migrating to Elasticsearch 5 in 4.0.0.

Performance #

Since we first migrated to Elasticsearch 1.x from MongoDB, we've learned a lot of hard lessons. With 4.0 and the upgrade to Elasticsearch 5, we took all those lessons and applied them to the new release and infrastructure setup. Here are some performance improvements and notes that came out of those implementations.

  • We've moved from monthly indexes to daily indexes. This change means that when you are filtering by the last 24 hours or last week, you are greatly reducing the amount of work Elasticsearch has to do to return the requested data.

  • Because of this, the average use case will see between 25% and 80% improvement to indexing throughput. This means that, with the same hardware, we can index the same documents much faster!

  • We moved from Groovy scripts to Painless scripts, which are four times faster. By moving to Painless scripts, we also greatly simplified the burden of having to modify configuration files to enable scripting!

  • Elasticsearch has added a lot of new data types since 1.7, and we are now taking advantage of them to reduce memory, storage and query costs. This means we can query the same data faster, using less memory doing it. We've also set up more sensible defaults to ensure we don't index very long strings.

  • We've also made several performance and reliability improvements to snapshots (backups).

Self Hosting improvements #

Our goal is to have everyone be able to setup and use Exceptionless in a matter of minutes. One of the areas that had hendered a lot of self hosters in the past was forgetting to update the elasticsearch.yml file, which is no longer an issue with the migration to Painless scripts. If you're a self hoster, check out the

  • The move to using Painless scripts as part of bulk updates and ingest pipelines reduced the burden of having to modify configuration files to enable scripting! Long gone are the times where you would have to reset your setup because you missed a configuration step.

  • We've also added various maintenance jobs that handle backups and restores automatically, so you don't have to worry about losing your data!

  • Future upgrades should be seamless as Elasticsearch now handles reindexing out of the box! Once you have migrated your data to Elasticsearch 5, we think future major upgrades will just be handled by the Exceptionless app itself!

  • You can now also use Docker or Azure arm templates to quickly set up a cluster. We really like this direction and will continue moving down this path, hopefully getting to the point where you can click a single button and have an Exceptionless production instance running locally!

Dashboards #

With Exceptionless 4.0.0, we continued to focus on finishing up backend improvements to both our repositories and parsers that we made in the 3.5 releases. We feel that all the pieces are finally in place to allow us to do custom dashboards in the near future, something we talk about in our 2017 Roadmap blog post.

Foundatio.Parsers now gives us the ability to define and validate custom aggregations using Lucene-style syntax. For example, lets say I want a date histogram that shows the min, max and average event value. I'd just need to pass date:(date min:value max:value avg:value) to the following endpoint: /api/v2/events/count?aggregations=date:(date min:value max:value avg:value) to return aggregations across a time series!

Upgrading to 4.0.0 #

The only users that need to worry about upgrading anything for this new release are self hosters. If you are self hosting Exceptionless, please review the Self Hosting Documentation, which contains information about upgrading your existing install.

Also, please take a look at the change log for a full list of the changes.

Always Improving #

We’re always striving to improve the efficiency of Exceptionless and all of our projects. If you see any room for improvement or have any comments when using anything from us, please send us an in-app message, submit a GitHub issue or contact us on the website.

And Lastly, Thanks! #

We'd like to say thank you to the community, project sponsors, and the Elasticsearch team for helping us ship 4.0.0. You guys rock!

Want to know what's coming next? #

Check out the 2017 Roadmap and let us know your thoughts on our planned development.

Setting Up New Elasticsearch Cluster, Reindexing & Mappings - Live Code Demo

Getting Ready for Exceptionless 4.0! #

In this live code review, we go over some of the recent changes and processes to get ready for Exceptionless 4.0, which include implementing a new Elasticsearch cluster, reindexing into it using the Elastic Arm templates, and updating existing events mapping.

The Elastic Arm templates automate the setup of an Elasticsearch with a few clicks, which is awesome.

Then, we had to update our existing events mapping to have a last updated field and is deleted flag so we can do incremental reindex passes.

WATCH NOW

Watch more Live Code Demos

2017 Exceptionless Feature, Functionality, and Enhancement Roadmap

exceptionless feature roadmap for 2017

Can you believe it's almost February already! Us either. No rest for the whicked, though, as we sit down to get cranking on our list of Exceptionless changes for 2017.

We have a a great list of features, functionality changes, and enhancements planned, and no doubt we'll work in a few other popular feature requests and, naturally, bug fixes along the way.

Let's take a look at the primary roadmap, shall we?

Upgrading to Elasticsearch 5 #

Upgrading to Elasticsearch 5 has been in the works for some time now, with early testing starting in late 2016. As of January 23rd, we have officially pushed the upgrade to production and are looking for feedback, missed bugs, and performance improvement reports from anyone using it!

Primary Elasticsearch 5 Benefits #

  • Elasticsearch 5 is the latest version of ElasticSearch and brings in massive improvements to event indexing speed, reduced memory sizes, and much more.
  • The Elasticsearch libraries support .NET Core, bringing us one step closer to fulfilling a vision that includes cross-platform support.
  • Our new implementation is built on a new, faster Microsoft Azure and SSD storage infrastructure. This will greatly increase the indexing and searching performance,  reducing dashboard latencies, among other things.
  • With the power of Foundatio.Parsers project, we have gained the ability to consume generic aggregations and pivot/report on data any way our users like it.
  • This upgrade also makes it easier to self host through the use of containers. Our goal is to move the entire app to use containers, which will allow you to self host in seconds!
  • We will now be doing daily indexes for more performance, fine grained backups, and the ability to more quickly change our indexes.

Future Exceptionless Notification Changes #

We're always working on notifications, we've got several distinct changes we want to work on in 2017. Here are a few of the important ones:

  • We will continue working on rich, rule based notifications for email, services like Slack and HipChat, and more.
  • Pausing notifications will soon be a thing!
  • You'll also be able to set up notifications based on the rate of events, so you can know when issues are ramping up or getting out of hand.
  • Lastly, you will be able to receive a periodic digest email based on activity.

Porting to .NET Core and Containers #

  • We want to port the Exceptionless app to .NET Core so we can run it anywhere (macOS, Linux or Windows).
  • Running on .NET Core also brings massive performance improvements and lower overhead, which will have positive trickle down affects throughout the app.
  • Once we support .NET Core, we can containerize the entire app, allowing us to scale more easily.
  • These changes will make self hosting super simple. You'll basically be able to run "docker run -d exceptionless`" and be done!

Custom Exceptionless Dashboards #

We know the dashboard is crucial for many things, and we want to make them more customizable to fit each user's individual cases. At the same time, we want to make sure we maximize business value. So, we plan on working on several features and functional aspects of dashboards, namely:

  • Creating pre-define dashboards that are more granular, such as device usage, breakdown by exception type, etc.
  • Allow users to create their own custom dashboards.

New Exceptionless Clients #

We really want to add new native clients to Exceptionless, and are including that as a major goal for 2017. Everything we are working on is moving in that direction, one way or the other, and we hope to make further announcements in this department soon!

We are, of course, always accepting pull requests as well!

What Are We Missing? #

What have we forgotten? We know everything isn't included on this list, but if we have missed something big that you can't live without, please let us know!

Otherwise, look for a new version announcement very soon, and get ready to upgrade and take advantage of several of these new upcoming changes.

Upgrading from Elasticsearch 1.x to 5.x - Live Coding Session

elasticsearch 1.x to 1.5 upgrade

New Weekly Coding Live Stream #

In the first episode our all new weekly live streaming demo session, Blake talks about the recent push to migrate Exceptionless from Elasticsearch 1.x to Elasticsearch 5.x.

One major part of this project has been updating Foundatio.Repositories and Foundatio.parsers, both of which are parts of our pluggable foundation block project for building distributed apps, to support Elasticsearch 5.x.

Once that was complete, we then worked on reindex scripts for migrating data from Elasticsearch 1.x to 5.x using Elasticsearch's external reindex.

Check out the live stream, and tune in weekly for the next episode.

Want Blake to demo something specific? Just leave a comment here and we'll make sure he gets it done!

WATCH NOW

Watch more Live Code Demos

2016 Recap - Let there be STATS!

Exceptionless 2016 in review

We were curious, so we thought we would put together some stats for 2016, along with and a recap of some of the notable changes and progression. Enjoy!

General Exceptionless Stats #

  • Over 120,000,000 events processed
  • 4716 users
  • 3614 projects
  • 2809 organizations

Release Stats #

Exceptionless Server: #

  • 7 releases
  • 88 watching
  • 774 stars
  • 236 forks

Exceptionless UI: #

  • 6 releases
  • 9 watching
  • 38 stars
  • 34 forks

Foundatio: #

  • 4 releases
  • 65 watching
  • 472 stars
  • 97 forks

Judging by the traffic to these posts and updates, they're all worth a read. Check them out!

Notable Changes / Milestones #

Server License Changes #

We changed the server licensing to use the Apache license. Now, every Exceptionless project is under the Apache license and there should be no more confusion on how it's licensed. Read more here.

Session Tracking & Management #

The ability to track and automatically manage a users session was added, giving you visibility into how long and what a user did while they were using your product. This also answers the question, "what did my users do leading up to this exception?" Read more here.

Fixed by Version #

In 2016, we also added the ability to mark an event as fixed in a specific version of your app. Read more here.

Performance & Reliability #

We spent a lot of time this past year making massive performance and reliability improvements across the board. Tons of bugs got fixed along the way, and additions like bulk updates and deletes, along with optimized query generation got squeezed in there, too! One big factor here is that it has allowed us to move to Elasticsearch 5 much quicker because all of these improvements allowed us to do things more generically throughout. Much less code to update/change!

Elasticsearch 5

Like we mentioned above, we've done a ton of work to ensure we can move to Elasticsearch 5:

  • Insanely fast document indexing and searching
  • Dynamic aggregations (allowing us to do completely customize stats on any data)
  • Reindexing support
  • Daily indexes
  • Making it easier to self host
  • Getting one step closer to server being able to run cross-platform

We have been doing QA on the Elasticsearch 5 migration since the end of the year and it is currently still in progress - coming soon!

Open Source ALL the Things #

We open sourced two major new libraries this year:

  • Foundatio.Repositories
    Foundatio.Repositories are generic repositories with an implementation for Elasticsearch
  • Foundatio.Parsers
    Foundatio.Parsers are Lucene query parsers that allow you to validate or modify Lucene queries with ease!

Other Noteworthy Enhancements #

Onward to 2017 - Stay Tuned for a Roadmap! #

We've got big things coming up, like Exceptionless 5.0 with Elasticsearch 5, which should boost performance gains like 80% faster indexing, lower memory usage, and native reindexing support! No slowing down over here, so stay tuned and enjoy the ride.

And, as always, if you have anything you'd like to see in upcoming Exceptionless releases, please let us know!

Exceptionless Helps Identify Issue Affecting 25,000 Users - Case Study

CEVO, Inc. LogoCase study time!

Today we've got a great example of Exceptionless helping software developers identify major issues in their web applications affecting thousands of users.

In this case, the problem was major enough that the development team stopped using WPF and rewrote their entire UI layer!

Here's what Will Graham, CEVO, Inc. developer, had to say when we asked him a few Exceptionless case study questions.

Exceptionless Case Study - CEVO, Inc. #

What’s the number one customer-facing bug Exceptionless helped you squash? #

"The issue was so prevalent we completely ditched WPF and rewrote our UI layer in GDI+/WinForms" - Will Graham, CEVO, Inc

Will:

Definitely crashes related to .NET Framework installation corruptions on end-user machines. While Exceptionless didn't (and can't) immediately point and say, "here's how to fix it", it did shine light on an issue that was affecting approximately 5% of our 500,000 installed base. The issue was so prevalent we completely ditched WPF and rewrote our UI layer in GDI+/WinForms, but Exceptionless gave us insight to see the problem and how many users were being affected.

Were you surprised at the initial results of using Exceptionless for the first time? How many errors were you seeing? #

"Exceptionless has changed our internal development process and how we approach code. Proper error handling and visibility is now a first-class priority for us and Exceptionless makes it super easy." - Will Graham, CEVO, Inc

Will:

Yes and no. Error handling and tracking had always been more of an afterthought - like, something is wrong, we need to look into it now. We weren't aware of issues until after our users reported it and filtering out the noise to find the signal of real issues was a time consuming process. Running on the Azure stack, we were frustrated with the information provided in the default diagnostics logging.

When we added Exceptionless, we knew we had problems with our system, we just didn't know where and to what extent. To be honest, Exceptionless has changed our internal development process and how we approach code. Proper error handling and visibility is now a first-class priority for us and Exceptionless makes it super easy.

What is the number one internal bug you were able to track down with Exceptionless? #

Will:

The biggest bug would have to be intermittent failures on our Service Bus instance. Using the metrics provided by Exceptionless, we were able to spend time implementing fault tolerance in very focused areas of the code based on incidence rate and end-user impact.

If you had one feature you’d like us to add to Exceptionless, what would it be? #

Will:

I'm not a huge fan of the current dashboard. It's effective, but with the amount of data Exceptionless has at it's disposal, I feel there's more interesting (and actionable) information that can be shown.

We Love It! #

Thanks for your time, Will! We love to hear from our users, and we'll definitely take that dashboard feedback and see what we can do there to make it more useful.

Do you have feedback on Exceptionless? Has it helped you find bugs and beat them into submission? Join us in the Exceptionless Discord and let us know!

Set Application Version for Improved Regression Notifications and Stacking

versioning(/assets/versioning.png)Do you get annoyed and overwhelmed by event and error notifications?

You probably have more than one version of your application running, and often older versions of your app may still be triggering events that have been fixed in newer versions.

**Those events and notifications aren't very **helpful, so we implemented a versioning system that allows you to set an application version on all events!

After setting an application version, when you mark an event fixed and give it a version number, it will only regress if it occurs again in a newer version of your app.

That means there is less noise for you to wade through, and you can focus on new issues in your application without seeing old or non-relevant events constantly.

How to Set an Application Version #

Setting an application version in Exceptionless is easy. By default, we attempt to resolve one automatically based on assembly attributes, but we recommend specifying one yourself for improved reliability and accuracy using the following examples.

.NET Version Specification Example #

using Exceptionless;
ExceptionlessClient.Default.Configuration.SetVersion("1.2.3");

JavaScript/Node.JS Version Specification Example #

exceptionless.ExceptionlessClient.default.config.setVersion("1.2.3");

Fixed! #

Great! Now, when you mark an error stack as fixed and enter the version that you fixed it in, that event stack will have a Fixed In [Version] tag and will only regress if it occurs again in a later version of your app. If it does regress, the stack then gets the REGRESSED tag.

Regressed(/assets/regressed.jpg)

Regressed

If you would like to view fixed events, you can always use the * wildcard or fixed:true in search.

We hope you find this feature useful, and as always don't hesitate to leave us feedback over on GitHub or by commenting below.

Real Time App Configuration and Event Settings with Exceptionless

exceptionless-project-settings-header

Bet You Didn't Know Exceptionless Could Do This... #

Have you ever needed to cut through the noise and just focus on one type of event (in real time), such as only error logs, to track down a bug?

Do you want to limit certain types of event reporting (in real time) to save your event quota and limit clutter?

What about controlling your application's settings or features in real time via Exceptionless, without having to update your files and deploy your app!?

Well, with our client configuration settings, you can do all that, and more, in real time, on a per-project basis via your Exceptionless dashboard!

How Could This Help Me? #

We'll talk details, below, but first lets look at a few scenarios where the above could be useful.

Scenario 1 - Way too many events! #

Let's say you're on the small plan, and you've got a bunch of warning log events clogging your system that you know about and are working on, but they are pushing you over your plan limits.

#BOOM - set a minimum log level of error, and Exceptionless won't report those warnings anymore and they won't count against your plan limits! This is a great way to get the most out of your Exceptionless plan.

Scenario 2 - This Authentication Issue is KILLING Me! #

Maybe you're having major issues with an authentication bug, but you've already set minimum log levels to only include errors. Well, now you also want to see the trace values for those events without opening the flood gates for every event by removing your minimum log level.

#NOPROBLEM - Just add a key for authentication that just lets trace events through!

Scenario 3 - You Said Something About Controlling My App's Features In Real Time? #

Yup! Our client configurations are basically just a key value pair dictionary, but what makes them powerful and helps them control your application's features is that they get updated in nearly real-time, meaning you can build settings, features, etc into your app that react to value changes, and if you change that configuration setting in Exceptionless, your app will react almost instantly!

This can be super useful, especially if changing your app's settings would normally require you to deploy to production. No need - just use Exceptionless!

Primer: How Project Settings Work #

First, settings updates do not count towards plan limits. We say that because each time an event is submitted, we send a response header with the current configuration version to the client, and if a newer version is available, it is retrieved and the latest configuration is applied. That means that config changes are nearly real time.

When the client is idle, we also check for config changes, including five seconds after client startup if no events are submitted at startup, and every two minutes after the last event submission.

If the version hasn't changed, nothing is retrieved, limiting data transfer, and no user information is ever sent when checking.

Turning Off Automatic Updating #

If you do not want the configuration settings to update when idle, you can turn off automatic updates. To do so, please follow the respective .NET or JavaScript/Node.js documentation.

The Main Event: Client Configuration #

client-configuration

Exceptionless client configurations are a dictionary of key value pairs that can be used to control the behavior of your app in real time by doing things like controlling data exclusions, protecting sensitive data, enabling and disable features, or disabling certain types of events (error, usage, log, 404, or session).

We also have some built in configuration key naming conventions (@@EVENT_TYPE:SOURCE) that the clients recognize for ignoring events based on event type and event source. Just replace EVENT_TYPE part with the event type (E.G., errorlog...) and the SOURCE (E.G., exception type or log source) you'd like the setting to apply to. Next, specify key value of false to discard matching events client side. It's worth noting that  log event types can also accept a log level value (E.G., TraceDebugInfoWarnError, or Fatal).

For example, we can use it to turn off all error events of type, lets say, System.ArgumentNullException, by using the key @@error:System.ArgumentNullException and the value false.

Or, we could turn off all error events entirely by using the * wildcard. So, the key would be @@error:* and the value would be false again.

Examples for the Above Scenarios #

Scenario 1

In scenario 1, above, we were trying to save our plan limits by limiting log events to only errors. So our Client Configuration key would simply be @@log:* with value Error.

Scenario 2

Here we already limited our log events to errors, but now we're troubleshooting a specific issue with authentication (let's say we're using the AuthController API), so we want to look at the trace messages coming through. We can override any general minimum log levels that we've defined by setting a level for a specific log source. So, all we would do is add the @@log:*AuthController key with value Trace! Then, when the bug's fixed, turn it off as needed.

Scenario 3

This is the cool one. Here you are wanting to, let's say, pass a value for a setting in your app that turns something on or off without having to re-deploy everything. This is super easy to accomplish all we need to do is create a setting which will control our feature! Let's assume we have have a feature flag to show a welcome screen. We will name this feature flag enableWelcomeScreen and create a new configuration setting respectively with a value of true (You can change this value at any time). These changes will be pushed based on the above "How Project Settings Work" section automatically, all we have to do is check the setting as shown below.

C#
using Exceptionless;
// Check the configuration settings for our enableWelcomeScreen feature flag with a default value of false.
if (ExceptionlessClient.Default.Configuration.Settings.GetBoolean("enableWelcomeScreen", false)) {
// Show the welcome screen!
}
JavaScript
// Check the configuration settings for our enableWelcomeScreen feature flag
if (exceptionless.ExceptionlessClient.default.config.settings['enableWelcomeScreen'] === true) {
// Show the welcome screen!
}

Pretty cool, right!

For more details on client configuration, check out the Client Configuration Project Settings documentation. Specific usage examples can be found on the .NET and JavaScript/Node.js documentation pages respectively.

Other Project Settings You Might Find Useful #

General #

general

If you go to Admin > Projects in Exceptionless, you can choose the project you would like to edit the settings for. Each project can have unique settings.

The default tab is "General," which simply houses the project name and attached organization. Nothing fancy here - pretty self explanatory.

API Keys #

exceptionless api keys

This tab is where you can generate an API key for your project. Again, pretty self explanatory. Hit "New API Key" and one gets generated. For more details on API usage, check out the API Usage documentation on GitHub.

Settings #

exceptionless project settings

This is where you can set data exclusions, customize error stacking, and build in some spam detection to your project.

Data Exclusions

There are several use cases where you might not want to send some data to your Exceptionless project. This field allows you to enter a comma delimited list of field names that will be removed and not transferred to Exceptionless. The perfect example here is a password field, or other personal and sensitive data.

The * wildcard is supported in this field and can be used at the end (password*), beginning(*password), or on both sides (*password*) of the field name to further customize your data exclusion.

Error Stacking

Control over error stacking is another level of customization project settings allows. You can specify user namespaces or common methods to ignore. More details below.

User Namespaces

Here you can enter a comma delimited list of namespace names that own your application. With those in place, the only methods that will be considered stacking targets are ones inside those namespaces.

Common Methods

If your code has shared utility methods that may generate a bunch of errors, this could be useful. Enter a comma delimited list of common method names that shouldn't be used as stacking targets, and they will be ignored.

Spam Detection

Spam is the worst. So, we added a "Spam Detection" list of common user agents that should be ignored, which you can add to as you see fit. This eliminates a lot of noise, and can be customized to help trim even more depending on your application.

Along with the comma delimited list of user agents to ignore, you can also tick the box that says "Reduce noise by automatically hiding high volumes of events coming from a single client IP address." This can ward off large numbers of events being submitted by a spammer or attack on your app.

Integrations

exceptionless integrations

Integrations with tools like Slack, HipChat, JIRA, Basecamp, and others are very popular and can add a level of automated notifications, etc, to your workflow. So, on the integrations tab of your project's configuration you can create web hooks to integrate with your service or others as mentioned. Each web hook has a URL that it can call, and options for when it should be called. When a selected event occurs, a POST request is submitted with either event or stack data in JSON format. For more details and sample data, visit the Exceptionless integrations documentation.

That About Covers It! #

We hope this will help you make the most out of your Exceptionless projects, allowing you to save some event submissions, get that customization you were looking for, etc.

Please let us know if you have any questions, comments, concerns, bugs, or anything else we can help with!