In the case study, we discuss how Elasticsearch let us improve our real-time dashboard, work with time zones more easily, index data much faster, not have to worry so much about pre-aggregating data, maintain much less code, improve backup and restoration, shrink disk space usage, and, perhaps most importantly, drastically improve scalability.
Basically, we really like Elasticsearch and you should check it out if you haven't already.
This week we want to talk about reference Ids and how we use them to improve customer service for Exceptionless. And how we think you can do the same for your users.
A reference id is a unique identifier, generated by the user, that allows you to look up a submitted event at a later time. This is important because event Ids are not created until after the event is processed, so there is no other way to look up an event. We also use Reference Ids to help deduplicate events server side, which is another reason why it’s important that they are unique.
One of the ways we use Reference Ids is to support end users, something that is very important to us. There is nothing more frustrating than being unable to help a user that is experiencing a problem because you don’t know what they are seeing (when there could be thousands of errors in your app).
To combat this issue, we always try to include a reference Id with every error message shown to the user. This allows end users to contact us with a Reference Id and receive help immediately for the error they are seeing because we can easily track down and view it.
The Exceptionless clients have built in support to generate and track unique Reference Ids for every error event. Behind the scenes, we register a default Reference Id plugin that sets a Reference Id on submitted error events and stores the Reference Id in an implementation of ILastReferenceIdManager. The Reference Id manager’s goal is just to persist and retrieve the last used Reference Id.
Since this is a default plugin, we can enable this behavior by calling the UseReferenceIds() method on the configuration object to register the default Reference Id on every error event.
Please note that you can create your own plugin to create your very own Reference Id(s).
To get the the last used Reference Id, you can call the GetLastReferenceId() helper method on the the ExceptionlessClient instance.
C#
usingExceptionless; // Get the last created Reference Id ExceptionlessClient.Default.GetLastReferenceId();
JavaScript
// Get the last created Reference Id exceptionless.ExceptionlessClient.default.getLastReferenceId();
You might have noticed how easy it is to get or add Reference Id’s to your events automatically. This makes it a breeze to let your developers track down user-facing issues by displaying the Reference Id to your end users.
We display Reference Ids to all of our end users anytime an error occurs in our ASP.NET WebAPI application. We accomplish this by adding a custom <a href="http://www.asp.net/web-api/overview/error-handling/web-api-global-error-handling" target="_blank">IExceptionHandler</a> and return a new error response to include the Reference Id as shown below:
var exceptionContext = context.ExceptionContext; var request = exceptionContext.Request; if(request ==null) thrownewArgumentException($"{typeof(ExceptionContext).Name}.{"Request"} must not be null",nameof(context));
// CreateErrorResponse should never fail, even if there is no configuration associated with the request // In that case, use the default HttpConfiguration to con-neg the response media type if(configuration ==null){ using(HttpConfiguration defaultConfig =newHttpConfiguration()){ return request.CreateResponse(statusCode, error, defaultConfig); } }
You might be thinking: "Reference Ids are great, but what do I do with them now that I have one." Well, you can view the event that they reference on our site or via our API. This can be accomplished two different ways:
Hotlinking: You can link directly to a submitted event by outputting a link in your UI or logs (e.g. https://be.exceptionless.io/event/by-ref/YOUR_REFERENCE_ID)
Search: You can search via our api/ui with reference:YOUR_REFERENCE_ID
We hope this article was helpful, and we'd love to know if you're using Reference Ids and how they've helped you help users, solve internal issues, etc. Post a comment below!
Prior to version 3.1, we used an early version of the Foundatio Jobs system to run our out-of-process jobs via Azure WebJobs. We found it to be quite a pain to figure out which jobs were running or eating up system resources because every job was titled Job.exe (just like figuring out the w3wp IIS process is running). Also, just to run an out-of-process job, one would have to compile the source, copy dependencies to a common bin folder, and then run an executable (Job.exe) with parameters that specify the job type.
These tedious and error-prone tasks that had to be completed just to get a job to run are a thing of the past.
Now, with Foundatio, our open source app building block solution used in Exceptionless, you just define a new Job that runs (via the run method) and you can use the Foundatio Jobs API to run the job in process, out of process, continuous, or one time without changing the implementation.
This new approach also gave us a great deployment strategy, for free. Simply copy the job executable and bin folders and run it anywhere!
Jobs (processes) running in Azure as an Azure web job #
Foundatio Jobs allows you to run a long running process (in process or out of process) with out worrying about it being terminated prematurely. By using Foundatio Jobs you gain all of the following features without changing your job implementation:
Run job in process
Run job out of process
Run job with a start up delay
Run job in an continuous loop with an optional interval delay.
In this sample we'll just define a new class called HelloWorldJob that will hold our job that increments a counter and derives from JobBase. Please note that there are a few different base classes you can derive from based on your use case.
But our goal is to run this out of process in an Azure WebJob (this also works if you want to run this as a service or from the desktop).
The first step is to create a new console application and reference the Foundatio NuGet Package and the project that contains our HelloWorldJob. We are going to call our console application HelloWorldJob. Inside of the Program class, we'll update the main method to run our job.
namespaceHelloWorldJob{ publicclassProgram{ publicstaticintMain(string[] args){ // NOTE: This should be the path to your App_Data folder of your website. var path = Path.Combine(AppDomain.CurrentDomain.BaseDirectory,@"..\..\..\Api\App_Data"); if(Directory.Exists(path)) AppDomain.CurrentDomain.SetData("DataDirectory", path);
// Get a service provider so we can create an instance of our job. var serviceProvider = ServiceProvider.GetServiceProvider("JobSample.JobBootstrappedServiceProvider,JobSample");
var job = serviceProvider.GetService<JobSample.HelloWorldJob>(); returnnewJobRunner(job,initialDelay: TimeSpan.FromSeconds(2),interval: TimeSpan.Zero).RunInConsole(); } } }
The last steps are to simply compile the project and deploy it to your Azure website!
Frank's back at it, adding support for collecting extra exception data that was already in the .NET client like module info, custom exception properties, and everything else that was already displayed in the UI when using the .NET client.
Read Configuration
We added support for reading configuration from environmental variables and app settings
**Closing Applications
** Closing applications after submission is now easier due to the new SubmittedEvent event handler.
**Custom Persistence Settings
** The new QueueMaxAttempts and QueueMaxAge configuration settings allow custom persistence settings and are intended to improve offline support. Thanks @airwareic!
Data Exclusions Improvements
We've made huge improvements to Data Exclusions, which now check default data and extra exception properties.
**New Overloads
** Thanks @xbelt for creating overloads for CreateLog and SubmitLog to accept the LogLevel enum!
Custom Themes
@mgnslndh updated the styling of the CrashReportDialog to play nice with custom themes. Thanks!
**Dependencies
** All dependencies (Nancy, NLog, etc) have been updated.
**Deprecated!
** The EnableSSL property has been marked Obsolete because it is no longer used. ServerURL is now being looked at for this.
If you thought Exceptionless was fast before, prepare to have your mind blown by what we've been able to do in version 3.1, which released today.
In short, we've reduced cpu and memory usage, increase caching efficiency, and sped up searching, all quite significantly as you'll see below.
Along with these speed boosts, we've also made Job improvements (look for a blog post on this soon) and upgraded to .NET 4.6 (self-hosters, please install .NET 4.6 on your servers before upgrading).
Details on the release changes can be found below.
We reduced the CPU and Memory load across the entire app. This allows the application to use fewer resources, meaning it has more time to process additional events, making everything faster. Between making Exceptionless and Foundatio 100% Async and these improvements, we've drastically increased the efficiency of the entire platform.
Below, we see the increase in performance from two examples. On the left, we see a reduction in CPU and Memory usage for a deployed web app instance. On the right is a visible reduction in CPU usage for an Elasticsearch node.
By profiling the Elasticsearch queries for efficiency and usage, we've been able to reduce the overall number we were running and improve the efficiency on the ones that still are.
Caching efficiency has been improved by removing redundant components that were utilizing valuable resources. For example, we removed the SignalR Redis Backplane, which drastically decreased the number of calls to Redis. Overall, we've made the app smarter throughout regarding how we cache and retrieve data.
We've offloaded long-running API tasks to background jobs, freeing up a lot of resources in the app and allowing us to scale work items independently. For example, marking a stack as fixed or removing a project may take a few moments to be updated now, but the trade-off is worth it. We're working on updating the UI experience to prompt users that the task is running in the background.
Exceptionless is now running on .NET 4.6, which has improved startup time due to various improvements with the new version. Self-hosting users should be sure to upgrade to .NET 4.6 on their servers before updating Exceptionless.
We're always striving to improve the efficiency of Exceptionless and all of our projects. If you see any room for improvement or have any comments when using anything from us, please send us an in-app message, submit a GitHub issue or contact us on the website.
We've been hard at work on several things here at Exceptionless, including a minor version release of our JavaScript Client!
Please see the details of this release below.
Also, we would like to give a shout out to @frankerbersoll, @srijken and the entire community for help squashing bugs, reporting issues, and providing general feedback. You guys rock.
Foundatio is a pluggable, scalable, painless, open source solution for caching, queues, locks, messaging, jobs, file storage, and metrics in your app.
In Version 3.0, we've made several improvements, including, as promised in our initial Foundatio blog post, going full async.
Take a closer look at the new enhancements, below, and head over to the GitHub repo to try Foundatio today. We think you'll find it very handy.
For all our current users, you'll need to upgrade your Foundatio NuGet package and existing Foundatio code to use the async implementations/conventions. The update should be fairly straightforward - we haven't had or heard of any issues so far, including when upgrading Exceptionless.
We've gone 100% async, which means no more blocking synchronous code, allowing your apps to be even more responsive. For more on async and its benefits, take a look at this Microsoft article.
Along with going completely async, we've removed non async methods from the API and simplified the API overall. We wanted to make it easy for developers to roll out new implementations and use the existing API.
New methods
We also added new, useful methods across all the APIs to make it easier to perform common tasks. One example includes adding CalculateProgress() to WorkItemHandlerJobs. We hope these helpers allow you to be more productive.
Because we always want more speed out of our apps (and we know you do, too), we used Red Gate's ANTS Performance Profiler to profile various apps, such as Exceptionless, and track down "hot" sections of code. By hot, we mean those pieces of code devoured more resources and took longer to run than all other application code. By doing so, we were able to pinpoint core pieces of Foundatio that were performing poorly compared to what they could be doing. Then we fixed them.
A few optimization examples
Maintenance code in various locations like queues, caches, etc, could and would run in a tight loop, sometimes pegging the CPU to 100% depending on the configuration. By running through and removing that maintenance code, or changing the way it was implemented to be triggered only when needed, rather than running in a constant worker loop, we drastically reduced CPU load and increased efficiency.
We also made massive improvements to queues and locks, on top of the maintenance code changes. Our Dequeue and AcquireLock methods would constantly pull for new items in a loop with a small timeout, which is obviously inefficient. Now we are using async monitors and pulse when a message is received over the message bus. This allows for huge improvements as the code will wait/sleep for a single monitor (of multiple monitors that could be waiting) to get triggered. This means your app isn’t wasting CPU cycles waiting, or external network calls looking for locks or new queue items. Instead, it’s running your app code and getting immediately notified when the lock is available or a new queue item is enqueued. Pretty slick!
Caching got a huge performance boost, too. For InMemoryCache clients, we moved from reflection-based object copying to using Expression Trees. This reduced the time required to get items from cache by a large percentage. Read more about implementing Expression Trees and see the difference it makes here.
We made many other improvements to ensure your apps run fast when using Foundatio, too many to be listed here. If you're already using it, just update your NuGet package to the latest version to take advantage of these improvements. If you're not using it yet, try it out. It's worth a shot, we promise!
If you missed our JavaScript Client announcement, this is a great chance to see just how quick and easy the client is to install and use! You can be finding those hard to track down bugs, errors, and exceptions, as well as tracking other events and log messages, in literally minutes.
We've made it super easy to get up and running, as you'll see below. Take a look and let us know what you think!
Log in to your Exceptionless Dashboard and get your project's API key (Admin > Projects > Select Project > API Keys)
* If you don't have a project, you'll need to create one by going to Admin > Projects > Add New Project
Open the extracted folder on your local machine and go to the "example" folder, then edit index.html.
Replace the API key in the exceptionless.js script tag with your own, and comment out the line with serverURL in it (not needed for demo).
Open index.html in your browser and open your browser's developer tools console so we can see the events fire.
* We've added quick-fire buttons to this page that automatically generate events for you to play around with for the purpose of this demo.
Click on one, generate 100 random events for instance, and you'll see the events queue up in the console. After a few seconds, you should see a queue processing notification telling you it's done processing.
Go back to your Exceptionless project dashboard, and you will see the events hit the system in real time.
* Make sure you are viewing the appropriate project and time period in your dashboard, otherwise you might not see the new events come in.
**Stop** beating your head against the keyboard tracking down those hard-to-find bugs and cut down your development time in general by using Exceptionless to **report and log errors, feature usages, and any other event** within your project.
We're not just for .NET developers anymore. Our JavaScript client allows you to use Exceptionless in almost **any development environment**, and you can get it up and running in minutes!
Learn more below, including how to get started in **just about a minute and a half**!
Now Serving... **5,000,000+ Events Per Month 2000+ Active Projects**
"Exceptionless is awesome. I've STOPPED worrying about whether I'm losing customers to bugs I don't know about." - Eric B.
This 1:37 video demonstrates how quick and easy it is to get the Exceptionless JavaScript client up and running. Take a few minutes to integrate it with your project and gain visibility you've never had before on bugs, feature usages, and more!
We've got a quick release for the Exceptionless JavaScript Client that includes a few new additions to what it supports and a quick bug fix or two.
The .NET Exceptionless client has received several improvements and some bug fixes in version 3.2. Details below!
If you thought Exceptionless was fast before, prepare to have your mind blown by what we've been able to do in version 3.1, which released today.
Foundatio is a pluggable, scalable, painless, open source solution for caching, queues, locks, messaging, jobs, file storage, and metrics in your app.
