Thinking and Talking

Data validation on REST API: Don’t trust clients input

mayworm — Sun, 29 Nov 2009 18:35:26 +0000

REST API’s have been becoming more and more common on Web applications/tools around the Internet, specially for those on open social context. It is also undebatable how the REST style architectures have been growing in enterprise applications, probably as a possible and more light-weight alternative for architecture to expose services, and/or produce a consumer-based application. REST solution are easy to visualize, specially because thinking in resource are on the most of the cases a natural way to think, very close to domains. Associating those things with the “simple”, fast and pragmatic way to implement a REST service, probably it is one of the principal reasons why so many applications in different scenarios have been adopting REST style in some way.

Although REST style architecture shouldn’t be so hard to define and implement(otherwise software developers would avoid it), there are several things that must be considered when adopting a REST solution, such as: modeling your resources, types, REST enterprise development, resource reuse, think about consumer-based services, data representation, apply standardization, URI strategy and the flexible programming model to complement it, and others. In additional to these things that should be considered, when go more deeply and start to design/implement the REST based-services/API, others concerns come up, for example error/notification messages and data input validation. It is important to keep in mind that probably the REST services will be consumed by severals different clients(or maybe not), and the quality/integrity data the services are receiving cannot be guarantee, different from a Web application(in most of the cases) where the presentation layer probably has a validator mechanism to be sure about what will be sent through(of course it doesn’t put away the validation moment into others layers). Taking it into account, a validation mechanism analyzing different possibilities becomes important on the REST API, guaranteeing that clients will be always notifying correctly instead of just get an “exception” back and have a way to keep track of the errors once they happen. When something unexpected happens(invalid data or system troubles for example) let the clients know that there is a problem or special situation that the system identified, notifying the clients through a correct message which must be wrapper in a previous known format and send back on the HTTP Response body following a specific HTTP code. Other important point is avoid risks on the service side having a safe code, understand the potential risks to the software and to make sure to have the appropriate mitigations in place, applying potential data validation mechanism to help with code quality too and don’t be affected for bad structured data comes from.

Going through the data validation input on REST, the REST API needs to ensure that this “layer” on the application is robust against all client’s input data, whether obtained from the external entities or known applications. Basically two approaches are useful(of course several others can be considered):

Integrity checks – ensure that data has not been tampered.
Validation – ensure that the data is strongly typed, correct syntax, within length boundaries, contains only permitted characters, or if numeric is correctly signed and within range boundaries, etc.

It is important to define the message error format, because the HTTP code associate with message error “envelop” is the way for REST API notify the clients in details about what happened, keep in mind that in a simple data input validation different types of validation rules can be broken, then a strategy to notify in a unique response all errors got during the process can be interesting. In additional to that to have a request ID in a error message help to keep track of the happenings in the logs.

The validation cab be performed on every tier as desired, however, on each layer should contains specifics validations trying to be complementary during the event process. Basically thinking in the REST “layer”, the validation approach could be:

Accept known good – Just the known data will be accepted and send through the layers. The data received through some data representation(for example: JSON/XML) must be parsed to the specific type and just the substantial fields must be checked for the event triggered. There are so many kind of interesting validator frameworks that can be “injected” into the code to validate the received data, following the domain definitions and keeping the data integrity between the layers.
Make clean – In an effort to have a “safe” input try to clean eliminating or removing not desired data.
No validation – Although this approach must be considered unsafe and strongly discouraged, specially because it brings a non defensive code and fragile services on the REST API, maybe the team wants to considered this way taking into account a totally secure and known environment and client applications, just pushing the input validation in a non explicit way for others “layers”, it means for example just the validation on database through constraints will be used.

And in this talk, the last but not least, is the checking URI, where depending of the method on the API and the URI form, specific validation can be necessary. It is important remember that most of the REST frameworks on different platforms provide mechanism to convert a “parameter” that forms a URI into a specific type, for example: /users/mary/project/55, the parameter project ID in this case “55″ will be convert to a long on the specific method responsible for that. But if the client application tried: /users/mary/project/55XYZ probably the client application will receive a error from the framework during the convert moment, instead of receive a message informing that the provided ID is not valid for a project. It means that adopt a strategy to receive all parameters as String and then check/valid it of them internally in the code, can provide the alternative to send correct/custom messages for the client application. It always depend how much details the software wants to take over. Still in this direction, for example the relation between “mary” and “project 55″ must be check, to guarantee that “mary” has rights on “project 55″ and can retrieve/edit data there. This simple case has been approaching here because in several cases the REST API is “resource integrity”, then the data just there is in a specific resource, where for example: /users/john/projects/55 is not allow, because project 55 is a “mary” project and ”john” cannot do nothing there, then a specific rule restrictions on the persistence layer must be in place.

Static Analysis: As part of reviewing what you are getting.

mayworm — Thu, 01 Oct 2009 22:15:19 +0000

Last week I came across with a friend that I was long time without see him, during some talks and beers he pointed something that caught my eyes. He told me about the new strategy adopted by the company where he works. Due to a market situation, they need to outsource some projects, or even some “parts” of projects. For that, they created contracts, rules, strategies for delivery, put a “upstream” process in place and so on. Ok, until this point nothing special or new. Once the vendors implement the requirements and these are accept, they will integrate those implementations on their code bases. And after that everybody will be happy, I mean, the vendor that provided a bundle of code and the company that has the system ready to support some market strategy, of course that I have been simplistic, because we know that this kind of process usually is not so smoothly specially during the beginning(probably middle and end).

But what caught my eyes during this talk was when I asked him about the rules on static testing, and he answered me “anh..humm.. we don’t have it in an agreement with the vendors”, then what I could see from him, they have good acceptance tests and specification-based tests, but a shy code-based testing. So if you think through a perspective of who will receive in some moment a bundle of code and integrate it in a code base and with time goes by become the responsible for that code received, of course that we should try to push some “quality tests” on it, otherwise we can just be adopting a “bomb” which will go off on the future. I’m not here saying that you need to be radical and push a strong control about cyclomatic complexity, even if it helps you to identify tests possibilities or improvements on code. But you can have a “modest” checklist and tuning rules on some static code analysis tools to point possibles spots on the code that you must use some static techniques before accept what the vendors delivered for you, as for example: software inspection, code review and tracing. You don’t need to bring bureaucracy, you can be “agile”, but careful with what you are getting from. It is worthwhile you identify what tools you need for static code analysis (there are good open source tools, or even good commercial tools), identify what you need from those tools, what exactly kind of rules you need, tuning the tools, keep a close relationship with the vendors because they are your partner on this static testing, and keep in mind that it is really useful, bringing quality and “explicitness” for the code and even for a better design.

So, with simple things in mind you can guarantee a code gift that you are getting, and probably a good relationship with vendors.

Overengineering: Now, who pays the price?

mayworm — Tue, 15 Sep 2009 18:25:06 +0000

Last week Felipe and I were talking about some situations that we came across, really similar situations, although we never worked before on the same company, we can see situations on software development around us that has the origin overengineering, specially the overengineering on detailed design stuff(I know that “design” is a huge topic, so only few aspects we considered in this talk). We started to discuss about what motived in that time the team to design a so complex solution for products, and why they forgot accommodating changes that is one of the most challenging points of good software design. Most of those softwares that we were discussing about are medium and small projects and non mission-critical application, but have complexity on the maintenance cycle. Of course that quality of a design depends on which constraints are satisfied early and which are left for later, and most of the time some constraints doesn’t have a right/fast answer. And as managing complexity is something that catches my eye(don’t forget that when projects do fail for reasons that are technical, in most of the cases it is often uncontrolled complexity), we decided to make a brainstorm analyzing different points on design, as economic factors, layers, patterns, concurrence, test, quality attributes, how the abstract ideas were addressed and if they tried to make a component-based design and how they used the object-oriented design. After our discussion(considering the projects that we talked), we got the following points, that if maybe considered for those designers, could reduce the problems that we came across:

On economic factors, we could see how designers have failed or been unable to anticipate future needs for change on an economic important issue that needs to be considered, that is the cost of making changes to a system. Specially the “Perfective maintenance”, that consist in extending and improving a system once it is operational, typically by providing new functionality/feature requested by users. For example, we could see designers “drawing” really complex UI solutions on top of MVC frameworks. But once the maintenance team receives new requirements that contains UI parts completely different about what was designed/thought, those new implementations become so hard and consume a lot time, where the price payed for a new “button” added on UI is unbelievable high.
We thought about layers several times during the talk and it remembered how many times we’ve seen a layers mix to cause overengineering in the projects and how it introduced a significant complexity during the moment on which the development team needed to say about the kind of change will be coming. We were surprised about the bundle of extra layers that we identify in some softwares that has to be maintained, but always continue addition anything in terms of value for the software design itself.
On patterns no question how they are important and lively for design process. I will assume that you are aware about architectural styles and design patterns. The enthusiasm of patterns adoption has sometime scarred me, taking a look in the design details is easy perceive that most of the time people are trying to find spots to push design patterns than using design pattern to solve a “common” problem that they came across. Other interesting point is the lack of care in creating documentation to express the reasons why those design patterns were adopted. Indeed, it can be argued that the enthusiasm for designing a better solution, these patterns has acted to expose the weaknesses in the “pattern concept adoption”. Then keep in mind the question of how useful it is to have an ever-increasing stock of patterns. The exaggerated mixed of patterns, arguing that it brings flexibility, probably it is not the right choice, because the complexity is coming together.
About documentation the team responsible for maintain may ﬁnd themselves with inadequate or inaccurate documentation, making their job hard to ensure that their changes are consistent with the original plans, and usually they are not, even becoming easy to introduce new bugs. And it is fact that the productivity is affected because when the team inherits an overengineering design, it is necessary to spend so much time learning the nuances of that design before the team can extend or maintain it, and probably the time constraint will be against you.
During the design moment, abstraction performs an important role, where comes metaphors and representations. We can perceive that representations provide abstraction of a system and they are linked to the idea of views. Some designers have been disregarding the detailed design, in this way, not providing the different views for the design, not allowing to see the software throughout different points of view. A good example of views is “4+1 View”, or even using the alternative forms to observe the design, as constructional, behavioral, functional and data-modeling. Then basically it means that the concept of a view, as a means of capturing a particular set of design properties, and as projected through the use of a representation.
Other interesting approach is keep the eyes on the construction planning, establishing the schedule to design(an of course debug and unit test), most of the time it tries to keep us “simpler”, specially when the project deviates from the schedule because we are spending more time on design that we “should”, and there are a lot chances of the reason is because the design is becoming hard/complex. The construction plan helps to organize the design(specially when project allocate more design activity to construction, usually on medium/small projects, allowing to flesh out details of the software design during the construction, and maybe it can naturally guides to a simpler design), define the order in which components are created and integrated and the software quality manage process, that directly weights on design quality. I know that is hard to know when the design is good enough, or how much details is enough or how much should be left to be done at the “typing moment”, and maybe these points helps on avoid overengineering.
On design for test, the suggestion is: keep in mind to design it to facilitate testing. It is really clear that when you design thinking on tests, result produced tends to be more formalized modules and classes, probably being more beneficial.

In additional, there is an interesting list of internal design characteristics that [McC04] mentions, and I like to have in mind:

Minimal complexity
Ease of maintenance
Loose coupling
Extensibility
Reusability
High fan-in
Low-to-medium fan-out
Portability
Leannes
Stratification
Standard techniques

I remember an important passage from [PW85], where Parnas and Weiss, in their active design review approach, point interesting properties of a design as: it should be well structured, simple, efficient, adequate (satisfying the requirements), flexible (easy to change), practical (module interfaces sufficient for the job), implementable, and standardized (documentation organized in a standard way).

We know that design is a heuristic process, which means that no solution is guarantee to work for all projects and probably doesn’t fit on all teams, then the idea was just write down a small subset of things to consider during approaches to design and try to focus on reduced-complexity benefit, the easier and safer the software will be to change and test. We know that simplicity is just a vision, and “maybe” we can try to find it keeping these things in mind.

- [PW85] D. L. Parnas and D. M. Weiss. Active design reviews: Principles and practices. In Proceedings of the Eight International Conference on Software Engineering, pp. 215-222, 1985.
- [McC04] S. McConnell, Code Complete: A practical Handbook of Software Construction, Microsoft Press, second ed., 2004.

Technical Debt: How much have you been paying for that?

mayworm — Tue, 11 Aug 2009 21:12:00 +0000

Long time ago Ward Cunningham brought the idea around the term “technical debt”[1]. After it others like Steve MaConnell[2] introduced more thoughts on this metaphor. It is quite interesting see this ongoing in a real scenario. And in additional to that to see the team getting used with the situation, defraying interest each release cycle, and more, without hope to amortize the debt during the software life time.

In a previous post I wrote about the “Quick-and-Dirty”[3] approach, where I put the phrase: “we need to keep in mind a way to clean it as time goes by.”. I guess it can explain what you must do after decide to acquire a debt, because you need to address some issues that in a specific moment are more important than the design/code quality. Situations like that can happen, for example time-to-market or even when stackholders push pressure.

I think that the technical debt can be in evidence on different spots(design, code, dependence management, and others), and this is really “fun” when the “software” decides to charge you. Just pay attention the moment before and after each delivery. Where are the stress points, and if they are continuous, I mean, the team are always complain about the same things. The test process is always messy and not easy to reproduce, build process is always broken and just one guy on the team knows about that, we don’t have a static code analyze tool in place, because otherwise it will show those several “poor” design/code that we built and we forgot to clean and we are saying most of the time: “throw away this software and go to implement from scratch”. When you are in this stage, your debt is really bad.

Now imagine this situation: you got a money from a bank, and just postpone to pay, when you think that you have the enough money you discover that the current debt amount is so high that you need at least to work one more year saving money to pay the debt. It is exactly what happens when you decide improve/refactor the design/code or poor solutions that you did a long long time ago. Then, if you take a technical debt, keep always looking back shortly to your stuff, otherwise it will be impossible to improve.

[1] – http://c2.com/doc/oopsla92.html
[2] – http://blogs.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspx
[3] – http://mmayworm.blogspot.com/2009/04/where-is-boundary-between-rapid-and.html

"Friends with Benefits": By REST

mayworm — Sat, 04 Jul 2009 08:13:00 +0000

All the time people are creating new Web applications and trying to push it on the market. Different ideas popup each few seconds. Some years ago most of the Web applications could live “alone”, I mean, without integration/interaction with others, but today? Can you imagine trying to create a “revolutionary”/cool Web application without a “friend approach”? Or even trying to transform your same old application in something on Web 2.0/3.0 without a “friend approach”? I guess no.

I mean around “friend approach” the idea about publishing what your Web application does or knows for others. It is exactly what the marshup idea brings, but it is quite interesting to see how this approach is so natural nowadays in most of people responsible to think in a solution for Web applications.

Looking through the point of view of who want to develop an application that will provide information(resource)/service, I have been taking a look how interesting is a REST Style architecture, of course that it is not my focus to judge about others approach for architectures, like SOAP, specially because each of them has advantages that depends of the context, scenario and several variables.

REST Style can go beyond of the architecture and naturally introduce the idea about client API for interact with your application, then REST provides a convenient bridge between the platform worlds and easier content streaming than SOAP, less work. Moreover, an application that is internet-based and the requirements for security is low, it is best implemented in REST.

Somethings that comes around REST Style:

Data representation with JSON/XML/ATOM.JSON has been adopting widely, where there are a bundle of API on different languages to deal with it, and how simple is to integrate it on JS objects on browser, for example on XMLHTTPRequest.
Resources are manipulated by HTTP verbs (GET, PUT, POST, DELETE).
Security, based on HTTP. But a common security/identity approaches like OpenID and OAuth has been adopting day in day out, in a simple way to work with tokens for example. Basically it allows your REST Style solution to adopt different kind of authentication providers(Like GMail for example).
Caching is an interesting part of a REST Style, and can be used by both server side and client side. On the server-side infrastructure can be plugged a cache mechanism to “intercept” and include the ability to deal with caching headers avoid consume resource on the server infrastructure and increasing response time to the client.

REST has proven itself to be simple, flexible and scalable. Depending the way that you address to implement the REST Style and the design you do, you can really add more advantages for your solution, taking in the possibility to spread easily the components that deals with the services/resources in a CDN way, getting performance, better response time trip and distributing.

More things than just find an "solution"

mayworm — Wed, 13 May 2009 07:58:00 +0000

During a daily talk through email, a friend of mine came to me with a question about an specific approach that he would like to adopt on a new application(information system type thing) that the company where he works will start soon. Basically the scenario is: They would like to build an UI in Delphi to “interact” with a Java backend running in a remote server, this interaction will happen practically for all UC’s that needs to deal with data and/or business rules, what means that it is not a single or isolated interaction. Due to they have a strong Delphi knowledge and they want to find an “opportunity” to make things in Java, they decided for this solution(The point here is not to question if they should or not start to make things in Java in this project). It is a medium size application in accordance with him.

In a superficial level, he showed two possibles solutions and asked my opinion about those. The two solutions were:

Implement a WS approach to integrate the Delphi’s UI and Java backend.
Implement a Java “client” that would run together with Delphi and would be the responsible to communicate with the Java remote services, where Delphi talks with this Java client through JNI.

Well, I confess that in a first moment, I wasn’t so “comfortable” with this kind of architecture solution, but I’m not in position to criticize them, because probably there were several motivations that drove them for this way (Delphi + Java), and the question that he brought was about the “bridge” between Delphi AND Java and not about if he should use Delphi OR Java. And of course both of them have their respective advantages for different solutions.

After sometime thinking about some possibilities for this “bridge”, I decided to write down some primordial questions related to his options, just to check if he was aware about “attached” things that can come with an architecture choice, things that can be expensive if the requirements changing comes. The questions were:

Assuming that you already wrote and implemented an use case to validate those possibilities, did you write and implement an use case to validate how hard can be apply a change in a code developed follow both choices?
If after some time effort you identify that you need to replace the current “bridge” solution for another one, were you planning to design a solution to protect the others parts of your software?
Did you analyze the possible costs of defects found in requirements time, architecture time, construction time, system test time or post-release time?
Did you implement a end to end equirement to prof that the solution is feasible?
Probably this design is complex, challenging or both for you. The development team is unfamiliar with the application area. Then, are you assuming to use an iterative approach for this development?

I was waiting some “no” as answers, but I was surprised when I got “no” for all question above. It just showed that sometimes the team is so focussed on a fast technical solution that they put in place the first one founded, and then probably they will forget to think about others things that is coming on the future.

Plugin Architecture: Reversed Effort

mayworm — Tue, 28 Apr 2009 19:06:00 +0000

Are you considering plug-ins in your next application/tool?

Plug-in architecture is not a new concept, probably as you know it comes from a long time, and there are a bundle of stable “solutions” to support that, such as “Eclipse/OSGi” and others. Indeed there are several good technical arguments that can catch the development team’s eyes when they decide to adopt this kind of solution, taking into account what they are planning for their application. For example:

How can a functionality be added in a future moment?
How can the functionality be increased after delivering?
How can activated a component at runtime, following a specific component/plug-in lifecycle?

There are some strategies/patterns/definitions behind the scenes when the talk is plug-in architecture. I really encourage you delve into this subject.

On the other hand, nowadays we can see interesting points from a business perspective , when a team decide to develop an application/tool on top of a plug-in architecture. It is fact that the market has been accepting these new applications/tool in a good way, not “just” because there are relevant good technical arguments (but again, it depends of what you are planing to develop in terms of application/tool), but because there are different benefits that can be seen through the business perspective.

Usually the business team can take advantage of the ecosystem created around of a plug-in architecture, and define a business model to support the application on the market, involving your costumers and patterns throughout a profit relationship. It is possible to do it in an easy way, because one possibility is introduced naturally:

The application can just create a “contract”, defining a functionality/opening that it does not provide itself, although must be added by plug-ins.

This interesting possibility opens the doors for actions that aggregate values for the business around the application/tool, even if it is a close or open license. These actions can be represented as:

The extensibility power, what brings for the costumers and patterns the sensation of “freedom”, where they want to improve/add function as they want, without your interaction.
The startup of a plug-in market, where third parties will put their mind to think, trying to create some interesting plug-in adding naturally value for your application, what really well worth. Even if they make money without involving yourself, they will be adding value for your business.
The feedback power, where your application/tool will be all the time being “touched” for others code, and comments/suggestions will popup.

Maybe we can call it as the reversed effort, where your application/tool will be really supported by the third parties, just because you provided a way for them to take part in your business. Of course that there are several others things additional on that, as for example the motivation of your application/tool and etc. But at least the plug-in architecture approach can bring/open more flexibilities and/or opportunities in terms of business, if it fits on your technical stuff.

Where is the boundary between "Rapid" and Quick-and-Dirty on software development?

mayworm — Thu, 16 Apr 2009 18:59:00 +0000

As you can see, it is a classical Dilbert comic strips, but that probably represent the truth in some spots, specially for those teams that just want to delivery the things, usually as a way to show up productivity. A potential problem here is when the “Quick-and-Dirty” approach becomes the rules and a lack of techniques, process, or even concern, put the “Dirty” as focus, postponing the “Rapid-and-Quality”.

I’m not here to say that “Quick-and-Dirty” is the evil, but I think that there are specifics moments to follow this approach and good ways to “clean” the code and artifacts during the way.

You need to take care, because “Quick-and-Dirty” is a kind of approach that seems to be comfortable(specially for maintenance context with the fast changes around the team), and it can spread through the team easily and sometimes creating a “group think“. It is easy to see as the detrimental to the quality of the software pop-ups just to satisfy the agility on the tasks resolution, and the firsts effects are deficient tests, documentation, control of software life cycle, and the low quality of the code.

If you change the perspective for the software maintenance, it becomes potentiality worst, where in a short time you can see “curatives” scattered throughout the software, without acceptable documentation or even unit tests to guarantee some kind of quality. A very common argument that people adopt to use this approach is: We are agile, We are rapid!. But please, don’t make a mistake, “Agile” in terms of software development approach has not the same meaning as “Quick-and-Dirty”, and even during the software development process if we decide to use some “Quick-and-Dirty” for some specific reason, we need to keep in mind a way to clean it as time goes by.

So, before you decide to dig into this approach or push it on your team, try to understand the potential side effects such as “bug and fix” process, that you will bring into the software environment, try to see if it is just a matter of organization, process or even good sense.

Linux Support for ECF Skype Provider

mayworm — Fri, 28 Dec 2007 16:40:00 +0000

Linux Support for ECF Skype Provider

ECF (2.0.0 Milestone 4a – N&N ) has the addition of linux support for the Skype provider, now it is possible connect to Skype 2.0 beta under Eclipse on Linux. The Skype Provider implements the ECF Call API, the Presence API, and the Datashare API.