Making the Move to Elasticsearch for Exceptionless 2.0

Screenshot 2014-09-03 15.03.16We know you've thought about it when using Exceptionless, and many developers have submitted formal requests. Search.

You need it, and we want to provide it. That's why we've decided to integrate Elasticsearch into 2.0, and it's going to be awesome!

Check out a few of the cool things we're going to be able to do with it, below, and let us know what you think!

What Elasticsearch Brings to the Table #

Simple scaleability #

Moving to Elasticsearch for event and stack storage will enable us to simplify many areas of the system, like stats, and also enable us to easily scale the service as it continues to grow.

Search ALL the things #

The new system will allow users to search by tags, date range, keywords, error type, and more, all while providing full statistics at the same time. This, specifically, has been our most requested feature, and we're glad to finally be able to say it's coming soon!

Faster, too! #

Elasticsearch removes the need for us to do stats incrementing for every type of view that we want to show. This simplifies the system, further increases performance, and improves throughput for the event collector.

Statistics everywhere #

You're not just going to get more accurate stats within Exceptionless, you're going to get them in real time, and we'll be able to display them in multiple time zones, too.

Future-proofing #

Instead of having to think of everything up front while we're building Exceptionless 2.0, Elasticsearch will enable us to provide new insights in the future, something every developer dreams of!

We Hope You're as Stoked as We Are #

Search has been a long time coming for Exceptionless, and we're super excited to be able to provide a robust, thorough, scaleable solution that will cover 99.99% of possible use cases. Hang in there - we're coding it in as fast as we can!

Case Study: Server Expiration Error for Pre-Alpha Game Release

game-postToday we've got a pretty cool case study that comes to us from a game development studio!

We love to see Exceptionless being used by companies with interesting projects and development pipelines, and what's more interesting than gaming, multi-server, and multi-player environments?

These guys also gave us some good feedback, which we'll address.

Check it out!

Project #

This user is working on a pre-alpha game that is, at the time of the feedback, only available to the in-house team. The game environment requires multiple servers that are deployed via script. When an old version of a server comes down and a new one is deployed, there is cleanup that must happen, etc.

"I like your product! I've previously written an exception-reporting system that did the same type of thing as Exceptionless but used a mail-server as the exception repository. Since that system was something I had to leave behind at my last company, I wanted to find a solution for my current company, and after comparing alternatives liked what your team had built, so that's what we're using now."

How Exceptionless Helped #

Because the game had such limited players, servers, and testing, it's tough to catch all the little bugs. Fortunately, Exceptionless was able to catch a potentially huge bug that would cause old versions of the servers that were still running to crash because of files being deleted by the new server development clean up scripts.

"If an old version was still running it would crash because its data-files got deleted. Since we're still in development mode there aren't enough people playing on our game servers to notice this exception - but we would have when going into alpha or beta test! Fortunately Exceptionless did notice and report this problem."

We think that's pretty awesome, and not just because we're huge nerds!

Feature Requests & Thoughts #

We were lucky enough to get some great feedback from these guys, as well.

Bug Ownership #

"We'd like a feature that allows our developers to claim ownership of bugs so that they're not seen by others in the Dashboard view by default."

With multiple developers on multiple projects, bugs can stack up and things can get messy. John doesn't need to see Billy's bugs from project A when he's working on his own bugs for project B.

We definitely agree here and understand, but there is a fine line between us being an error reporting service and getting into bug tracking type features. We had a previous product that tried to do too much and turned people off so we really wanted to try and keep Exceptionless simple. That being said, we want to make Exceptionless integrate with other apps much more in the future and make it really easy to create new integrations. We're working on this now with Exceptionless 2.0, coming soon!

Multiple Services on Single Server #

"One other thing that was painful for me personally: we run multiple services of the same type on a single server (e.g. multiple instances of "game-server.exe"). In order to ensure that each server has its own queue folder and logfile, I had to write a chunk of custom code."

Again, we totally agree! The client in Exceptionless 2.0 will be MUCH simpler and will make things much easier. You will be able to easily use in-memory storage and be able to plug in different storage implementations.

We Love Feedback! #

If you're a current user, we'd love to hear how you've used Exceptionless to cut down on bugs and build better apps. If you've got any criticisms or feature feedback/requests, keep those coming as well - they help us improve!

Have an awesome day!

Upcoming Exceptionless Version 2.0 Overview & Review

Many of you guys have been following along in previous posts as we went through the details of the new features and enhancements coming with the next version of Exceptionless, V2.0.

To make things easier for everyone, we thought we'd do an overview post as well, so all the new features can be found in one location and people can click to read more on the specific ones they are interested in.

Exceptionless 2.0 Features & Enhancements #

Event Based Reporting System #

Post Event ExceptionlessMany users have asked for ways to use Exceptionless to report additional types of events, rather than just errors. With version 2.0, we are moving to an event based system that will accommodate such requests.

Read full article...

Simplified API #

Exceptionless API Documentation2.0 will have a new, manageable API with tons of great documentation and examples. Take a look at the preliminary documentation at the below link, and make sure to give us any feedback you might have.

Read full article...

A Pluggable System #

Pluggable SystemThe new pluggable system will allow customization and translation throughout the Exceptionless platform, including integration with third-party services and more. Read on for more details and source code about event parsing, the event pipeline, and formatting.

Read full article...

Client Rewrite #

new-client-headerThe Exceptionless client has been completely rewritten to be highly simplified and extensible, work with Mono and Project K, include additional platform specific clients, and much more. Take a look at an extended event data usage example at the link below.

Read full article...

New Message Bus & Queueing #

In an effort to improve scalability, allow for new functionality to easily be added to Exceptionless, make the system less coupled, process things more efficiently, go fully Async, and further support Azure, we’ve been working hard on a new message bus and queueing system.

Read full article...

Job System Enhancements #

A lot of time has been spent on enhancing the job system behind Exceptionless. From the ability to run jobs standalone, making testing easier, to more ways to run jobs and the ability to auto-scale jobs based on resource constraints, lots of great changes are coming!

Ready full article...

Feedback Welcome! #

If you've taken a look at the upcoming features and have any comments, please let us know! We're working on everything as fast and hard as we can, and will hopefully have an ETA to have everything finished soon.

New Price Plans for Medium to Large Teams/Projects

We've had quite a few requests for pricing plans that support users, retention, and errors somewhere between our old medium and enterprise plans, so we crunched the numbers and came up with new large and extra large plans that find a happy medium.

Both plans include more users, more retention, and more errors. Check out the details on each package below or view the Pricing Page for a quick summary.

Free #

The free plan, which is great for trying things out or super small personal projects, includes provisions for one project, one user, three days of error retention, and three thousand errors per month.

It does not include premium features such as advanced notifications, web hooks, and other future premium features.

Small - $15 #

For $15 per month, you get five projects, ten users, thirty days of retention, 15,000 errors per month, and premium features.

Medium - $49 #

The medium Exceptionless plan offers fifteen projects and twenty-five users. You get ninety days of retention, 75,000 errors per month, and premium features.

Large Plan - $99 (new) #

The large plan is new, and includes unlimited projects and users, 180 days of error retention, 250,000 errors per month, and premium features.

This is quickly becoming a popular plan and makes sense for a lot of medium to medium-large teams and projects.

Extra Large - $199 (new) #

The extra large plan was created to bridge the gap between large and enterprise by quadrupling the number of errors per month to 1,000,000. You still get unlimited projects and users, as well as 180 days of retention and premium features.

Enterprise - $499 #

For those businesses and enterprise-level teams that have several large projects and need to handle huge numbers of errors, the Enterprise plan offers everything that the Extra Large plan offers, with 3,000,000 errors per month.

Custom #

And, of course, should you need to customize a plan to handle even more errors, we can always get you taken care of, just let us know!

Job System Enhancements - Version 2's Got Em!

Summer means vacations and pool time, but we haven't stopped working on Exceptionless 2.0. Things are coming along nicely, and today we're here to talk about the job system and the code being written to enhance it.

After you read this article, check out the previous V2.0 feature and detail articles, if you haven't already. Good stuff in there!

  1. Exceptionless 2.0 – In the Making
  2. Event Based Reporting System
  3. Simplified API
  4. A Pluggable System
  5. Exceptionless 2.0 Client Rewrite Sneak Peak and Example
  6. New Message Bus and Queuing System
  7. Job System Enhancements

Job System Enhancements #

Standalone #

Jobs can easily be run standalone now, which makes it much easier to test the system. You won't have to worry about your application pool shutting down prematurely and killing your job half way through it's long-running work item.

More Ways to Run #

With 2.0, you'll be able to run jobs in process, as a service, as a standalone exe, or as Azure WebJobs. Cool, huh?

Need More? Run More! #

If you need to process more tasks, simply fire up more jobs. This will save resources and money when using Azure WebJobs, as you can auto-scale jobs based on resource constraints.

We use jobs to send emails, call web hooks, process events, and much more, so these enhancements will allow for better resource control when scaling, and in general.

Questions? #

We've thrown out a lot of information about Exceptionless 2.0 and all the new features, enhancements, and tweaks that it's going to get. We'd love to hear from some of the regulars out there and see if we've missed anything obvious. Check out the links to the other articles at the top of the page and let us know. Thanks!

Version 2.0's New Message Bus and Queueing Systems

In an effort to improve scalability, allow for new functionality to easily be added to Exceptionless, make the system less coupled, process things more efficiently, go fully Async, and further support Azure, we've been working hard on a new message bus and queueing system.

Lets take a look at a few of the details surrounding these new systems we're building for Exceptionless 2.0. Take a look and let us know what you think. If you've got questions or comments, we'd love to hear them!

Message Bus #

The new message bus allows us to publish and subscribe to messages across all our resources. We subscribe to different types of messages to send SignalR notifications to the client.

Queueing #

The new queueing system allows us to enqueue expensive tasks that can be handled at a later time. This lets us greatly reduce the processing and latency times of the api.

  • We stream event data that is posted to the event controller directly into the queue without taking the IO or Memory hit of processing it. This means that we can process more errors, faster, with less resources.

The system also supports retrying and discarding of data.

  • We queue emails that need to be sent, as well as user defined webhooks that need to be called with data. Email servers on the sending and receiving can go offline or error out while sending, but by queuing the notification emails we can ensure you always get them by re-sending in the future, after a failure occurs. In the event that we can't send you an email after a few retries, we can discard the notification.

More on Version 2.0 #

If you're just now learning about the upcoming Exceptionless 2.0, make sure to catch up on previous feature announcements and examples by reading the below articles.

Exceptionless 2.0 Client Rewrite Sneak Peek Usage Example

new-client-header

As Exceptionless 2.0 continues to become a reality, we thought we would give everyone a little taste of what you will be able to do with the new, rewritten client. Continue reading for a glimpse at the primary features, along with a complete usage example for adding extra data to events.

After you check it out, let us know if you have questions or suggestions. We're listening!

New Client Features #

  • The Exceptionless client has been completely rewritten to be highly simplified and extensible.
  • Will work with Mono and Project K.
  • The base client is PCL, and we will have platform specific clients that add additional functionality for each platform.
  • Adding extra data to events is extremely easy.

View Client Source

Extended event data usage example #

<img loading="lazy" class="aligncenter size-large wp-image-9100" src="/assets/img/news/ex-client-1024x420.png" alt="Exceptionless Code Example" width="940" height="385" data-id="9100" srcset="/.png 1024w, /assets/img/ex-client-300x123/nt.png 1167w" sizes="(max-width: 94/

Lets break that example down, line by line, shall we? Check out the client source if you want to take a look at the complete code.

First, set your API key.

var client = new ExceptionlessClient(config => {
config.ApiKey = "API_KEY_HERE";

Then, send events to your own free Exceptionless server install.

config.ServerUrl = "https://exceptionless.myorg.com";

Now, read config settings from attributes.

config.ReadFromAttributes();

Read config settings from a config section in your app/web.config.

config.ReadFromConfigSection();

Store all client data including the offline queue in the store folder, by default isolated storage is used.

config.UseFolderStorage("store");

Exclude any form fields, cookies, query string parameters, and custom data properties containing "CreditCard".

config.AddDataExclusions("CreditCard");

Add the "SomeTag" to all events.

config.DefaultTags.Add("SomeTag");

Add the "MyObject" custom data object to every event.

config.DefaultData.Add("MyObject", new { MyProperty = "Value1" });

Add a custom event enrichment that will add a tag called "MyTag" to every event.

config.AddEnrichment(ev => ev.Tags.Add("MyTag"));

Register a custom log implementation that uses NLog.

config.Resolver.Register(new NLogExceptionlessLog());

The Startup method is specific for each platform and wires up to all relevant unhandled exception events so that they will be automatically sent to the server.

client.Startup();

Manually catch and report an error with a custom tag on it.

try {
throw new ApplicationException("Boom!");
} catch (Exception ex) {
ex.ToExceptionless().AddTags("MyTag").Submit();
}

Let users add their email address and a description of the error.

client.UpdateUserEmailAndDescription(client.GetLastReferenceId(), "me@me.com", "It broke!");

Create and submit a log message and add an extra "Order" object to the event.

client.CreateLog("Order", "New order created.")
.AddObject(new { Total = 14.95 }, name: "Order")
.Submit();

Submit a feature usage event that will let you see how much certain features of your app are being used.

client.SubmitFeatureUsage("FeatureA");

Submit a page not found event so you can keep track of your broken links and fix them.

client.SubmitNotFound("/badpage");

Listen to all events being sent and cancel any errors that are "IgnoredType".

client.SubmittingEvent += (sender, args) =>
args.Cancel = args.Event.IsError() && args.Event.GetError().Type.Contains("IgnoredType");

Settings data is synced in real-time with the project settings in your Exceptionless project on the server.

client.Configuration.Settings.Changed += (sender, args) =>
Trace.WriteLine(String.Format("Action: {0} Key: {1} Value: {2}", args.Action, args.Item.Key, args.Item.Value));

You can use those settings to control behavior in your app.

if (client.Configuration.Settings.GetBoolean("IncludeMyCustomData", false))
Trace.WriteLine("Should include my custom data");

That's All There Is To It! #

After checking out the above example, we hope you agree that we've drastically simplified and improved the process of adding data to events, allowing for much more flexibility.

As always, if you have any questions, comments, suggestions, or concerns, let us know!

Read more about Exceptionless 2.0 #

Coming in Exceptionless 2.0 - A Pluggable System

Pluggable System

In the last Exceptionless 2.0 article, we announced the upcoming simplified API. Today, we want to introduce another major piece of V2.0 - the pluggable system.

Plugins will allow customization and translation throughout the Exceptionless platform, including integration with third-party services and more. Read on for more details about pluggable details such as event parsing, event pipeline, and formatting.

Event Parsing #

  • Has access to the raw POST data as well as the content type and submission client info.
  • Used to translate that raw data into events.
  • Can easily create new plugins to support new data formats like system logs.
  • Can be used to support other JSON formats like adding support for clients made for other systems.

Source

Event Processor #

  • Can be used to add new functionality to the system.
  • Gets called on startup, when an event is starting to be processed and when an event is done being processed.
  • Has access to settings from both the org and project level.
  • Can be used to create integrations for 3rd party services like HipChat, Trello, GitHub, Slack, etc.

Source

Formatting #

  • Used to control how events are displayed in the system.
  • Controls the summary view of an event.
  • Controls the stack title.
  • Controls what notification emails look like.
  • Controls which view is used to display the details of an event.

Source

We believe building a pluggable exception reporting system and allowing third-party service and app access will create one of the most flexible, usable, and friendly solutions on the market.

Coming Soon #

We're anxious to get Exceptionless 2.0 wrapped up, but we do not have an ETA currently. We are working hard and making good progress, so keep an eye out for more sneak peeks, feature announcements, and progress reports!

As always, please let us know if you have any feedback or questions.

More from the Upcoming Exceptionless 2.0: Simplified API

Exceptionless 2.0 API SimplifiedSince going open source, we've wanted to simplify the API and make it easier to work with.

We're taking the time to do it now, and it's going to be awesome!

Exceptionless 2.0, coming soon, will have a new, manageable API with tons of great documentation and examples. Take a look at the preliminary documentation at the below link, and make sure to give us any feedback you might have.

API Simplified #

  • New REST API documentation and samples site.
    Take a look and let us know what you thinkExceptionless API Documentation
  • Event POSTs take the raw data and use a plugin system to interpret that data and translate them into events.
    • This allows us to take literally any data and turn it into events in the system.
    • The POST data is captured as a raw bytes and added immediately added to a queue for processing.
    • Plugins can easily be created to support new data formats like system logs.
  • This simplified API will make creating libraries for other platforms dead simple.
  • The API lives in a separate project and can be hosted on high-performance systems like the new Helios IIS host.
  • Makes it easy for us to migrate the UI to a SPA app.
  • Now uses OAuth 2.0 in addition to supporting API tokens.
  • Highly consistent REST API modeled after GitHub and Stripe.
  • It's so simple you can just use CURL as a client.

We hope you're as excited as we are to have this new, improved, more complete, and more usable documentation. Stay tuned for more details on the upcoming Exceptionless 2.0, and don't forget to leave a comment letting us know what you think.

Exceptionless 1.5 Released!

Exceptionless Version 1.5While we're on the march to Exceptionless 2.0, we're still making updates and fixing bugs on version 1. Today, we'd like to announce that Exceptionless 1.5 has been released, which includes several server changes and bug fixes, as well as major client code base optimization.

Please update your client to version 1.5 and take a look at the other changes and bug fixes, below. We've done quite a bit of work to notifications, added throttling to improve coverage on small and free plans, and improved performance in a few places.

Server Changes #

  • Added throttling to accounts that are over their usage limits. If an account is sending a high number of errors, the errors will be throttled on an hourly basis so that the entire plan limit won’t be used up immediately. This allows for a distributed sampling of the errors instead of only capturing everything in a short period of time.
  • Added a site notification that shows you when error submissions are being throttled or if you are over your monthly plan limits.
  • Removed total count from most recent errors list as it was a very expensive to calculate while providing little value.
  • Fixed a bug with notifications that could cause some users to get spammed. Now notifications only send a maximum of 10 notifications per project every 30 minutes.
  • Greatly simplified the authentication logic for the web api pipeline.
  • Added the ability to print all content on the error occurrence page.
  • The pager will no longer scroll to the top of the current list when changing pages.
  • Updated the paged lists to only refresh the list data via push notifications when you are on the first page.
  • The list data will only be updated in real time if the data matches the current filter criteria.
  • Fixed a bug where the loading indicators would appear on the suspended and manage organization pages.
  • Fixed a bug where the save button on the manage organization page would have improper styling.
  • Fixed a bug where a HttpAntiForgeryException could be thrown when accessing the website.
  • Fixed a bug where a ArgumentException would be thrown if multiple model validation errors occurred on a single page.
  • Fixed a bug where a NullReferenceException could be thrown when signing up.
  • Added some additional checks to try and resolve the user profile when an invited user signs up.
  • Fixed a bug where an updated organization notification could be sent before the user was authorized to access the organization.
  • Fixed a bug where empty OS Name and Version values were being shown in the errors environment section even if they didn't exist. This could happen if the client was reporting from an azure website instance.
  • Changed billing plans to use per month error limits.
  • Fixed a bug where the BillingManager could throw a NullReferenceException for a newly added organization. This could happen because the primary node had not replicated the content to the secondary nodes or the data wasn't cached on creation.
  • Updated various MongoDB collections to not persist empty array fields.
  • Fixed a bug where some cache entries were not automatically expiring.

Client Changes #

It's highly recommend that you update your clients to 1.5 as we did major optimizations to the client code base.

  • Greatly simplified how the client processes and sends errors. The client now properly handles the various status codes that can be returned from the service.
  • Added an event that allows you to customize the request object before it is sent to the service.

As always, please let us know if you have any questions!