Elasticsearch Case Study - By Exceptionless

Elasticsearch case study

We recently wrote a case study regarding how Elasticsearch has helped us improve speed and ease of scale as we've continued to improve Exceptionless.

Last week, Elasticsearch published that case study on their blog, so we wanted to make sure our readers had a chance to check it out!

We've written about our use of Elasticsearch in numerous articles before, including:

In the case study, we discuss how Elasticsearch let us improve our real-time dashboard, work with time zones more easily, index data much faster, not have to worry so much about pre-aggregating data, maintain much less code, improve backup and restoration, shrink disk space usage, and, perhaps most importantly, drastically improve scalability.

Basically, we really like Elasticsearch and you should check it out if you haven't already.

Do you use Elasticsearch? #

Let us know how it's helped you, or if you have any questions we might be able to answer, by commenting below!

Using Reference Ids for Better Customer Service

Exceptionless Reference IDs

This week we want to talk about reference Ids and how we use them to improve customer service for Exceptionless. And how we think you can do the same for your users.

What's a Reference Id? #

A reference id is a unique identifier, generated by the user, that allows you to look up a submitted event at a later time. This is important because event Ids are not created until after the event is processed, so there is no other way to look up an event. We also use Reference Ids to help deduplicate events server side, which is another reason why it’s important that they are unique.

Using Reference Ids for the Greater Good #

One of the ways we use Reference Ids is to support end users, something that is very important to us. There is nothing more frustrating than being unable to help a user that is experiencing a problem because you don’t know what they are seeing (when there could be thousands of errors in your app).

To combat this issue, we always try to include a reference Id with every error message shown to the user. This allows end users to contact us with a Reference Id and receive help immediately for the error they are seeing because we can easily track down and view it.

Reference Id Example #

The Exceptionless clients have built in support to generate and track unique Reference Ids for every error event. Behind the scenes, we register a default Reference Id plugin that sets a Reference Id on submitted error events and stores the Reference Id in an implementation of ILastReferenceIdManager. The Reference Id manager’s goal is just to persist and retrieve the last used Reference Id.

Since this is a default plugin, we can enable this behavior by calling the UseReferenceIds() method on the configuration object to register the default Reference Id on every error event.

C# Example #

using Exceptionless;
ExceptionlessClient.Default.Configuration.UseReferenceIds();

JavaScript Example #

exceptionless.ExceptionlessClient.default.config.useReferenceIds();

Please note that you can create your own plugin to create your very own Reference Id(s).

To get the the last used Reference Id, you can call the GetLastReferenceId() helper method on the the ExceptionlessClient instance.

C#
using Exceptionless;
// Get the last created Reference Id
ExceptionlessClient.Default.GetLastReferenceId();
JavaScript
// Get the last created Reference Id
exceptionless.ExceptionlessClient.default.getLastReferenceId();

You might have noticed how easy it is to get or add Reference Id’s to your events automatically. This makes it a breeze to let your developers track down user-facing issues by displaying the Reference Id to your end users.

We display Reference Ids to all of our end users anytime an error occurs in our ASP.NET WebAPI application. We accomplish this by adding a custom <a href="http://www.asp.net/web-api/overview/error-handling/web-api-global-error-handling" target="_blank">IExceptionHandler</a> and return a new error response to include the Reference Id as shown below:

public class ExceptionlessReferenceIdExceptionHandler : IExceptionHandler {
public Task HandleAsync(ExceptionHandlerContext context, CancellationToken cancellationToken) {
if (context == null)
throw new ArgumentNullException(nameof(context));

var exceptionContext = context.ExceptionContext;
var request = exceptionContext.Request;
if (request == null)
throw new ArgumentException($"{typeof(ExceptionContext).Name}.{"Request"} must not be null", nameof(context));

context.Result = new ResponseMessageResult(CreateErrorResponse(request, exceptionContext.Exception, HttpStatusCode.InternalServerError));
return TaskHelper.Completed();
}

private HttpResponseMessage CreateErrorResponse(HttpRequestMessage request, Exception ex, HttpStatusCode statusCode) {
HttpConfiguration configuration = request.GetConfiguration();
HttpError error = new HttpError(ex, request.ShouldIncludeErrorDetail());

string lastId = ExceptionlessClient.Default.GetLastReferenceId();
if (!String.IsNullOrEmpty(lastId))
error.Add("Reference", lastId);

// CreateErrorResponse should never fail, even if there is no configuration associated with the request
// In that case, use the default HttpConfiguration to con-neg the response media type
if (configuration == null) {
using (HttpConfiguration defaultConfig = new HttpConfiguration()) {
return request.CreateResponse(statusCode, error, defaultConfig);
}
}

return request.CreateResponse(statusCode, error, configuration);
}
}

The next step is to replace the existing IExceptionFilter with the one above.

Config.Services.Replace(typeof(IExceptionHandler), new ExceptionlessReferenceIdExceptionHandler());

Finally, when an error occurs in your app, you’ll get a more user friendly error response that contains a Reference Id!

{
"message": "An error has occurred.",
"reference": "411085622e"
}

We’ve found that this allows consumers of our API to quickly contact us with a reference id and get up and running quickly!

Looking up Events by Reference Id #

You might be thinking: "Reference Ids are great, but what do I do with them now that I have one." Well, you can view the event that they reference on our site or via our API. This can be accomplished two different ways:

  • Hotlinking: You can link directly to a submitted event by outputting a link in your UI or logs (e.g. https://be.exceptionless.io/event/by-ref/YOUR_REFERENCE_ID)
  • Search: You can search via our api/ui with reference:YOUR_REFERENCE_ID

Your Turn! #

We hope this article was helpful, and we'd love to know if you're using Reference Ids and how they've helped you help users, solve internal issues, etc. Post a comment below!

A Better Approach to Running Azure WebJobs

Azure Webjobs

Lets talk about jobs in the Exceptionless world for a minute and how you can use our methods to improve your Azure WebJobs.

A job is a specific task/process that runs and does something like send a mail message, etc.

Out with the Old #

Prior to version 3.1, we used an early version of the Foundatio Jobs system to run our out-of-process jobs via Azure WebJobs. We found it to be quite a pain to figure out which jobs were running or eating up system resources because every job was titled Job.exe (just like figuring out the w3wp IIS process is running). Also, just to run an out-of-process job, one would have to compile the source, copy dependencies to a common bin folder, and then run an executable (Job.exe) with parameters that specify the job type.

These tedious and error-prone tasks that had to be completed just to get a job to run are a thing of the past.

In with the New #

In Exceptionless 3.1 we focused on refining and improving jobs. To do so, we created a new console application for each job and specified settings in the code versus error prone command line options as shown here.

Now, with Foundatio, our open source app building block solution used in Exceptionless, you just define a new Job that runs (via the run method) and you can use the Foundatio Jobs API to run the job in process, out of process, continuous, or one time without changing the implementation.

This new approach also gave us a great deployment strategy, for free. Simply copy the job executable and bin folders and run it anywhere!

Jobs (processes) running in Azure as an Azure web job #

Exceptionless Jobs and Processes

How you can implement a better Azure WebJob #

Foundatio Jobs allows you to run a long running process (in process or out of process) with out worrying about it being terminated prematurely. By using Foundatio Jobs you gain all of the following features without changing your job implementation:

  • Run job in process
  • Run job out of process
  • Run job with a start up delay
  • Run job in an continuous loop with an optional interval delay.

In this sample we'll just define a new class called HelloWorldJob that will hold our job that increments a counter and derives from JobBase. Please note that there are a few different base classes you can derive from based on your use case.

using Foundatio.Jobs;

public class HelloWorldJob : JobBase {
public int RunCount { get; set; }

protected override Task<JobResult> RunInternalAsync(JobRunContext context) {
RunCount++;
return Task.FromResult(JobResult.Success);
}
}

Now that we have our job defined we can run our job in process with a few different options:

var job = new HelloWorldJob();
await job.RunAsync(); // job.RunCount = 1;
await job.RunContinuousAsync(iterationLimit: 2); // job.RunCount = 3;
await job.RunContinuousAsync(cancellationToken: new CancellationTokenSource(TimeSpan.FromMilliseconds(10)).Token); // job.RunCount > 10;

But our goal is to run this out of process in an Azure WebJob (this also works if you want to run this as a service or from the desktop).

The first step is to create a new console application and reference the Foundatio NuGet Package and the project that contains our HelloWorldJob. We are going to call our console application HelloWorldJob. Inside of the Program class, we'll update the main method to run our job.

using System;
using System.IO;
using JobSample;
using Foundatio.Jobs;
using Foundatio.ServiceProviders;

namespace HelloWorldJob {
public class Program {
public static int Main(string[] args) {
// NOTE: This should be the path to your App_Data folder of your website.
var path = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, @"..\..\..\Api\App_Data");
if (Directory.Exists(path))
AppDomain.CurrentDomain.SetData("DataDirectory", path);

// Get a service provider so we can create an instance of our job.
var serviceProvider = ServiceProvider.GetServiceProvider("JobSample.JobBootstrappedServiceProvider,JobSample");

var job = serviceProvider.GetService<JobSample.HelloWorldJob>();
return new JobRunner(job, initialDelay: TimeSpan.FromSeconds(2), interval: TimeSpan.Zero).RunInConsole();
}
}
}

The last steps are to simply compile the project and deploy it to your Azure website!

Questions? #

If you have any questions please feel free to contact us via our contact page, in app message, GitHub issues or Discord.

Exceptionless.JavaScript 1.2 Release Notes

Exceptionless JavaScript ClientWe've got a quick release for the Exceptionless JavaScript Client that includes a few new additions to what it supports and a quick bug fix or two.

Shout out to @frankebersoll for his help!

Please download and update to the source code here, and view the full change log here.

JavaScript Client v1.2 Notes #

Errors #

@frankebersoll added support for deduplicating JavaScript errors. Thanks!

Data Collection #

Frank's back at it, adding support for collecting extra exception data that was already in the .NET client like module info, custom exception properties, and everything else that was already displayed in the UI when using the .NET client.

Node Info #

Support for collecting Node module info has also been added (thanks again, @frankebersoll).

Bug Fix - Invalid States #

An issue where Data Exclusions could cause events to be submitted in an invalid state has been fixed.

As always, let us know if you have any questions or feedback! #

Exceptionless.NET 3.2 Release Notes

Exceptionless LogoThe .NET Exceptionless client has received several improvements and some bug fixes in version 3.2. Details below!

A special shout-out to @airwareic, @srijken, @xbelt, and @mgnslndh for contributing and helping make this release happen. You guys rock!

Please download and update to the latest source here, and you can view the full change log here.

.NET Exceptionless Client v3.2.0 Release Details #

Additions & Improvements #

Read Configuration
We added support for reading configuration from environmental variables and app settings

**Closing Applications
** Closing applications after submission is now easier due to the new SubmittedEvent event handler.

**Custom Persistence Settings
** The new QueueMaxAttempts and QueueMaxAge configuration settings allow custom persistence settings and are intended to improve offline support. Thanks @airwareic!

Data Exclusions Improvements
We've made huge improvements to Data Exclusions, which now check default data and extra exception properties.

**New Overloads
** Thanks @xbelt for creating overloads for CreateLog and SubmitLog to accept the LogLevel enum!

Custom Themes
@mgnslndh updated the styling of the CrashReportDialog to play nice with custom themes. Thanks!

**Dependencies
** All dependencies (Nancy, NLog, etc) have been updated.

**Deprecated!
** The EnableSSL property has been marked Obsolete because it is no longer used. ServerURL is now being looked at for this.

Fixes #

**Startup()
** Startup() was overriding configured dependencies - Fixed!

**Empty Errors
** We fixed a bug where you could have empty errors submitted with no stack trace info.

API Keys

  • Previously set valid API keys were being overwritten with default API keys, so we fixed it.
  • We also fixed an issue where ApiKey couldn't be changed after the client was initialized.

Reference IDs
An issue with submitting generated reference IDs was resolved, thanks to @srijken

**Updating WebAPI Package
** @srijken also fixed another issue where updating the WebApi package would remove the Exceptionless Module. Thanks again!

NLog
Nlog wasn't working with .NET 4.0. This has been resolved.

IsolatedStorage
There was a problem that caused IsolatedStorage to not be able to be used. Fixed!

Min Log Level
NLog and log4net have been updated to allow setting the min log level.

What Say You? #

As always, we're listening to your feedback, comments, suggestions, and rants!

Exceptionless 3.1 Released! If You Thought It was Fast Before...

exceptionles-3-1-headerIf you thought Exceptionless was fast before, prepare to have your mind blown by what we've been able to do in version 3.1, which released today.

In short, we've reduced cpu and memory usage, increase caching efficiency, and sped up searching, all quite significantly as you'll see below.

Along with these speed boosts, we've also made Job improvements (look for a blog post on this soon) and upgraded to .NET 4.6 (self-hosters, please install .NET 4.6 on your servers before upgrading).

Details on the release changes can be found below.

It's All About Speed! #

CPU & Memory #

We reduced the CPU and Memory load across the entire app. This allows the application to use fewer resources, meaning it has more time to process additional events, making everything faster. Between making Exceptionless and Foundatio 100% Async and these improvements, we've drastically increased the efficiency of the entire platform.

Below, we see the increase in performance from two examples. On the left, we see a reduction in CPU and Memory usage for a deployed web app instance. On the right is a visible reduction in CPU usage for an Elasticsearch node.

cpu-memory-percentage-improvements

Elasticsearch Queries #

By profiling the Elasticsearch queries for efficiency and usage, we've been able to reduce the overall number we were running and improve the efficiency on the ones that still are.

search-request-rate-improvements

Caching #

Caching efficiency has been improved by removing redundant components that were utilizing valuable resources. For example, we removed the SignalR Redis Backplane, which drastically decreased the number of calls to Redis. Overall, we've made the app smarter throughout regarding how we cache and retrieve data.

cache-improvements-2

Caching-improvements

Long-running API Tasks #

We've offloaded long-running API tasks to background jobs, freeing up a lot of resources in the app and allowing us to scale work items independently. For example, marking a stack as fixed or removing a project may take a few moments to be updated now, but the trade-off is worth it. We're working on updating the UI experience to prompt users that the task is running in the background.

Other Improvements #

Jobs #

We've made each Job a console app, so it's much easier to debug and deploy. Look for a full blog post on Job improvements soon.

.NET 4.6 #

Exceptionless is now running on .NET 4.6, which has improved startup time due to various improvements with the new version. Self-hosting users should be sure to upgrade to .NET 4.6 on their servers before updating Exceptionless.

Upgrading #

For full release notes and to download the latest version, please visit the GitHub Release Page.

Always Improving #

We're always striving to improve the efficiency of Exceptionless and all of our projects. If you see any room for improvement or have any comments when using anything from us, please send us an in-app message, submit a GitHub issue or contact us on the website.

Exceptionless JavaScript Client V1.1.0 Release

Exceptionless JavaScript ClientWe've been hard at work on several things here at Exceptionless, including a minor version release of our JavaScript Client!

Please see the details of this release below.

Also, we would like to give a shout out to @frankerbersoll, @srijken and the entire community for help squashing bugs, reporting issues, and providing general feedback. You guys rock.

Exceptionless JavaScript Client V1.1.0 Release Notes #

You can view the full change log for a list of all changes in this release.

You can also Download Exceptionless JavaScript Client V1.1.0 on the official release page.

Introducing Foundatio 3.0! Now with Async & Increased Efficiency

Foundatio 3.0Foundatio is a pluggable, scalable, painless, open source solution for caching, queues, locks, messaging, jobs, file storage, and metrics in your app.

In Version 3.0, we've made several improvements, including, as promised in our initial Foundatio blog post, going full async.

Take a closer look at the new enhancements, below, and head over to the GitHub repo to try Foundatio today. We think you'll find it very handy.

For all our current users, you'll need to upgrade your Foundatio NuGet package and existing Foundatio code to use the async implementations/conventions. The update should be fairly straightforward - we haven't had or heard of any issues so far, including when upgrading Exceptionless.

What's New in Foundatio 3.0 #

API Changes #

We've gone 100% async, which means no more blocking synchronous code, allowing your apps to be even more responsive. For more on async and its benefits, take a look at this Microsoft article.

Along with going completely async, we've removed non async methods from the API and simplified the API overall. We wanted to make it easy for developers to roll out new implementations and use the existing API.

New methods

We also added new, useful methods across all the APIs to make it easier to perform common tasks. One example includes adding CalculateProgress() to WorkItemHandlerJobs. We hope these helpers allow you to be more productive.

Efficiency Improvements #

Because we always want more speed out of our apps (and we know you do, too), we used Red Gate's ANTS Performance Profiler to profile various apps, such as Exceptionless, and track down "hot" sections of code. By hot, we mean those pieces of code devoured more resources and took longer to run than all other application code. By doing so, we were able to pinpoint core pieces of Foundatio that were performing poorly compared to what they could be doing. Then we fixed them.

A few optimization examples
  • Maintenance code in various locations like queues, caches, etc, could and would run in a tight loop, sometimes pegging the CPU to 100% depending on the configuration. By running through and removing that maintenance code, or changing the way it was implemented to be triggered only when needed, rather than running in a constant worker loop, we drastically reduced CPU load and increased efficiency.
  • We also made massive improvements to queues and locks, on top of the maintenance code changes. Our Dequeue and AcquireLock methods would constantly pull for new items in a loop with a small timeout, which is obviously inefficient. Now we are using async monitors and pulse when a message is received over the message bus. This allows for huge improvements as the code will wait/sleep for a single monitor (of multiple monitors that could be waiting) to get triggered. This means your app isn’t wasting CPU cycles waiting, or external network calls looking for locks or new queue items. Instead, it’s running your app code and getting immediately notified when the lock is available or a new queue item is enqueued. Pretty slick!
  • Caching got a huge performance boost, too. For InMemoryCache clients, we moved from reflection-based object copying to using Expression Trees. This reduced the time required to get items from cache by a large percentage. Read more about implementing Expression Trees and see the difference it makes here.

Check It Out - Feedback Appreciated! #

We made many other improvements to ensure your apps run fast when using Foundatio, too many to be listed here. If you're already using it, just update your NuGet package to the latest version to take advantage of these improvements. If you're not using it yet, try it out. It's worth a shot, we promise!

Once you've given it a go, please let us know what you like, or what you hate, by posting a issue on Foundatio's GitHub Repo.

Until next time, happy coding!

Find Software Bugs with the Exceptionless JavaScript Client in Minutes

Exceptionless JavaScript Client
If you missed our JavaScript Client announcement, this is a great chance to see just how quick and easy the client is to install and use! You can be finding those hard to track down bugs, errors, and exceptions, as well as tracking other events and log messages, in literally minutes.

We've made it super easy to get up and running, as you'll see below. Take a look and let us know what you think!

Report & Log Errors & Events in 1:37 #

The Text Version #

A simple step-by-step guide to getting the Exceptionless JavaScript Client demo up and running on your local machine.

The only pre-requisite is an Exceptionless account, or you can host the entire platform yourselfOpen source, baby!

  1. Download and extract the Exceptionless JavaScript Client from GitHub
  2. Log in to your Exceptionless Dashboard and get your project's API key (Admin > Projects > Select Project > API Keys)
    * If you don't have a project, you'll need to create one by going to Admin > Projects > Add New Project
  3. Open the extracted folder on your local machine and go to the "example" folder, then edit index.html.
  4. Replace the API key in the exceptionless.js script tag with your own, and comment out the line with serverURL in it (not needed for demo).
  5. Open index.html in your browser and open your browser's developer tools console so we can see the events fire.
    * We've added quick-fire buttons to this page that automatically generate events for you to play around with for the purpose of this demo.
  6. Click on one, generate 100 random events for instance, and you'll see the events queue up in the console. After a few seconds, you should see a queue processing notification telling you it's done processing.
  7. Go back to your Exceptionless project dashboard, and you will see the events hit the system in real time.
    * Make sure you are viewing the appropriate project and time period in your dashboard, otherwise you might not see the new events come in.

It's That Easy! #

As you can see, the JavaScript client is super easy, super fast, and super flexible, for use in almost any project you might have.

We hope you'll give it a shot, and we also hope you'll let us know what you think by contacting us via an in-app message, submitting an issue on GitHub, or through the website.

Here's to becoming Exceptionless!

Squash Bugs Faster & Speed Up Development with Exceptionless

Developer Bug Tracking

Calling All Developers!

**Stop** beating your head against the keyboard tracking down those hard-to-find bugs and cut down your development time in general by using Exceptionless to **report and log errors, feature usages, and any other event** within your project.

We're not just for .NET developers anymore. Our JavaScript client allows you to use Exceptionless in almost **any development environment**, and you can get it up and running in minutes!

Learn more below, including how to get started in **just about a minute and a half**!

Now Serving...
**5,000,000+ Events Per Month
2000+ Active Projects**

"Exceptionless is awesome. I've STOPPED worrying about whether I'm losing customers to bugs I don't know about." - Eric B.

Up & Running In Minutes #

This 1:37 video demonstrates how quick and easy it is to get the Exceptionless JavaScript client up and running. Take a few minutes to integrate it with your project and gain visibility you've never had before on bugs, feature usages, and more!

IT'S. THAT. EASY. #

"Exceptionless allows us to prioritize fixes smarter by showing frequently occurring errors. In real time!" - Ryan H.

The only thing you need to get started is a free Exceptionless account, or you can even host it yourself! Once you have that, you can user the JavaScript or .NET client to get started pushing events to your new dashboard.

Then you'll be on your way to becoming... Exceptionless.